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Abstract 


The  cognitive  radio  field  currently  lacks  a  standardized  test  methodology  that 
is  repeatable,  flexible,  and  elfective  across  multiple  cognitive  radio  architectures. 
Furthermore,  the  cognitive  radio  field  lacks  a  suitable  framework  that  allows  testing  of 
an  integrated  cognitive  radio  system  and  not  solely  specific  components.  This  research 
presents  a  cognitive  radio  test  methodology,  known  as  CRATM,  to  address  these  issues. 
CRATM  proposes  to  use  behavior-based  testing,  in  which  cognition  may  be  measured  by 
evaluating  both  primary  user  and  secondary  user  performance.  Data  on  behavior-based 
testing  is  collected  and  evaluated.  Additionally,  a  unique  means  of  measuring  secondary 
user  interference  to  the  primary  user  is  employed  by  direct  measurement  of  primary  user 
performance.  A  secondary  user  pair  and  primary  user  radio  pair  are  implemented  using  the 
Wireless  Open- Access  Research  platform  and  WARPLab  software  running  in  MATLAB. 
The  primary  user  is  used  to  create  five  distinct  radio  frequency  environments  utilizing 
narrowband,  wideband,  and  non-contiguous  waveforms.  The  secondary  user  response  to 
the  primary  user  created  environments  is  measured.  The  secondary  user  implements  a 
simple  cognitive  engine  that  incorporates  energy-detection  spectrum  sensing.  The  effect  of 
the  cognitive  engine  on  both  secondary  user  and  primary  user  performance  is  measured  and 
evaluated. 


IV 


Table  of  Contents 


Page 

Abstract .  iv 

Table  of  Contents .  v 

List  of  Figures . viii 

List  of  Tables .  x 

List  of  Acronyms .  xi 

I.  Introduction .  1 

1.1  Problem  Statement .  2 

1.2  Contributions .  3 

1.3  Overview .  4 

II.  Related  Work .  5 

2.1  Cognitive  Radio  Definitions  .  5 

2.1.1  Mitola,  1999  .  6 

2.1.2  Kantor,  1999  and  Thomas,  2005  .  6 

2.1.3  Cognitive  Radio  Networks .  7 

2.1.4  Haykin,  2005  .  7 

2.1.5  IEEE  802.22  and  DySPAN .  8 

2.1.6  Eederal  Communications  Commission .  9 

2.1.7  Wireless  Innovation  Eorum .  9 

2.1.8  DARPA  XG  Program .  10 

2.1.9  Cognitive  Radio  Definitions  Summary .  10 

2.2  Architectures .  10 

2.2.1  Spectrum  Sensing,  Sharing,  and  Cooperation .  13 

2.2.2  Topology .  13 

2.2.3  Cognitive  Engine .  14 

2.2.4  Miscellaneous  Eeatures .  14 

2.2.5  IEEE  802.22  Architecture .  15 

2.2.6  AEIT  Cognitive  Radio  Eab  Architecture .  15 

2.3  Testbeds .  17 

2.3.1  Emulation-based  Testbeds .  18 

2.3. 1.1  CMU  Emulator  .  19 


V 


Page 

2.3. 1.2  WHYNET .  19 

2.3. 1.3  BEE2 .  19 

2.3. 1.4  ORBIT  and  MINT .  20 

2.3. 1.5  CREW .  20 

2.3. 1.6  DYSE .  20 

2.3.2  Non-Emulation  Testbeds .  22 

2.4  Performanee  Metries .  22 

2.4.1  Speetrum  Sensing  .  22 

2.4.2  Avoiding  Causing  Interferenee .  23 

2.4.3  Cognitive  Radio  Artifieial  Intelligenee .  25 

2.4.4  Communieation  Performanee  .  25 

2.5  Benehmarks .  25 

2.6  Cognitive  Radio  Test  Methodologies .  26 

2.6.1  Radio  Environment  Map  Seenario  Driven  Testing  Approaeh  ....  26 

2.6.2  Psyehometrie  Approaeh .  27 

2.7  Cognitive  Radio  Prototyping  Platforms .  28 

2.8  Cognitive  Radio  Speetrum  Paradigms  .  29 

III.  Proposed  Test  Methodology  .  30 

3.1  Overview .  30 

3.2  Cognitive  Radio  Behavior-Based  Evaluation .  31 

3.3  Performanee  Metries .  33 

3.4  Benehmarks .  34 

3.5  Means  of  Testing .  38 

3.6  Test  Eramework .  38 

3.7  Comparison  to  the  Eiterature .  39 

3.7.1  Key  Charaeteristies .  39 

3.7.2  Results .  40 

3.8  Summary .  40 

IV.  Methodology  . 43 

4. 1  Problem  Baekground .  43 

4.1.1  Goals  and  Hypotheses .  43 

4.1.2  Approaeh .  44 

4.2  System  Boundaries .  45 

4.3  System  Serviees .  46 

4.4  Workload .  47 

4.5  Performanee  Metries .  49 

4.6  System  Parameters . 49 

4.7  Workload  Parameters .  51 


VI 


Page 


4.8  Factors .  54 

4.8.1  System  Factors .  54 

4.8.2  Workload  Factors .  55 

4.9  Evaluation  Technique .  55 

4.10  Experimental  Design .  58 

4.11  Methodology  Summary .  59 

V.  Results  and  Analysis  .  60 

5.1  Evaluating  Behavior  Based  Cognitive  Radio  Testing  .  60 

5.2  Effect  of  Cognitive  Engine  on  Secondary  User  Throughput  and  BER  ....  61 

5.3  Effect  of  Cognitive  Engine  on  Primary  User  Throughput  and  BER . 70 

5.4  Scoring .  76 

5.5  DYSE .  78 

VI.  Conclusion  .  79 

6.1  Summary .  79 

6.2  Euture  Work .  80 

Appendix  A:  Data  Validation .  82 

Appendix  B:  Spectrum  Sensor  Core .  96 

References . 104 


vii 


List  of  Figures 


Figure  Page 

2.1  Cognitive  Radio  Cognition  Cyele .  12 

2.2  AFIT  Cognitive  Radio  Lab  Arehitecture  System  Funetional  Diagram .  16 

2.3  Spectrum  Overlay  and  Underlay  Paradigms .  29 

4.1  DS A  Testbench  SUT .  46 

4.2  PU  Created  Spectrum  Holes  .  48 

4.3  Cognitive  Engine  Algorithm  Flowchart .  52 

4.4  PU  Created  Workload  Environments .  56 

4.5  Experimental  Setup .  57 

5.1  PU  and  SU  Throughput,  Environments  1  and  2  .  62 

5.2  PU  and  SU  Throughput,  Environments  3  and  4  .  63 

5.3  PU  and  SU  Throughput,  Environment  5 .  64 

5.4  Secondary  User  BER .  69 

5.5  Primary  User  BER .  75 

A.  1  Histograms  of  Total  Experiment  Elapsed  Time .  84 

A. 2  Total  Experiment  Elapsed  Time  Residuals .  85 

A. 3  Scatterplot  of  Total  Number  of  Transmissions  Received .  86 

A. 4  Residuals  of  Eitted  Einear  Model:  PU  Total  Number  of  Transmissions .  87 

A. 5  Residuals  of  Pitted  Einear  Model:  SU  Total  Number  of  Transmissions .  89 

A. 6  Scatterplot  of  Symbols  Counted  .  90 

A. 7  Histogram  of  Spectrum  Sensor  Thresholds .  91 

A. 8  Histogram  of  Mean  Received  Signal  Strength  (RSS) .  91 

A. 9  Experimental  Data  for  Throughput  and  BER .  93 

A.  10  Experimental  Data  for  Throughput  and  BER .  94 

viii 


Figure  Page 

B.l  Spectrum  Sensor  Functional  Block  Diagram .  97 

B.2  Spectrum  Sensor  Top-Level  Schematic . 100 


IX 


List  of  Tables 


Table  Page 

2. 1  Cognitive  Radio  Definitions  .  11 

3.1  Example  CRATM  Benchmark .  37 

3.2  Comparison  Between  Cognitive  RAdio  Test  Methodology  (CRATM)  and 

Published  CR  Test  Methodology  Attributes  .  41 

4.1  SUT  Factors .  54 

5.1  Secondary  User  Throughput .  65 

5.2  Relative  Comparison  of  Secondary  User  Throughput  with  CE  On .  67 

5.3  Secondary  User  BER .  68 

5.4  Primary  User  Throughput .  71 

5.5  Relative  Comparison  of  Primary  User  Throughput  with  CE  On . 72 

5.6  Primary  User  BER  .  74 

5.7  Performance  Scoring,  SU  Weight=50%  PU  Weight=50% .  76 

5.8  Performance  Scoring,  SU  Weight=25%  PU  Weight=75% .  77 

A.  1  SU  Waveform  Configuration  Eegend .  83 

A. 2  Fitted  Einear  Model  Parameters  .  92 

B. l  Spectrum  Sensor  Area  Summary . 102 

B.2  Spectrum  Sensor  Timing  Summary . 103 


X 


List  of  Acronyms 


Acronym 

Definition 

ACRO 

AEIT  Cognitive  Radio  Eaboratory 

AI 

artifieial  intelligence 

BER 

bit  error  rate 

BW 

bandwidth 

CE 

eognitive  engine 

CN 

cognitive  network 

CR 

eognitive  radio 

CRATM 

Cognitive  RAdio  Test  Methodology 

CRN 

eognitive  radio  network 

CRS 

eognitive  radio  system 

CUT 

eomponent  under  test 

d-OEDM 

dis-eontiguous  OEDM 

DSA 

dynamie  spectrum  access 

DYSE 

dynamie  spectrum  environment  emulator 

DySPAN 

Dynamie  Speetrum  Aeeess  Networks 

EET 

East  Eourier  Transform 

ECC 

Eederal  Communieations  Commission 

EPGA 

field  programmable  gate  array 

ESK 

frequeney  shift  keying 

OEDM 

orthogonal  frequeney  division  multiplexing 

OODA 

Observe-Orient-Deeide-Aet 

OSA 

Opportunistie  Speetrum  Access 

PER 

packet  error  rate 

XI 


Acronym 

Definition 

PU 

primary  user 

QoS 

Quality  of  Service 

REM 

radio  environment  map 

RF 

radio  frequeney 

RKRL 

Radio  Knowledge  Representation  Language 

SDR 

software  defined  radio 

SPEC 

Standard  Performanee  Evaluation  Corporation 

SER 

symbol  error  rate 

SU 

seeondary  user 

SUT 

system  under  test 

USRP 

Universal  Software  Radio  Peripheral 

WARP 

Wireless  Open-Acess  Researeh  Platform 

xii 


A  TEST  METHODOLOGY  EOR  EVALUATING  COGNITIVE  RADIO  SYSTEMS 


I.  Introduction 

First  introduced  in  1999,  cognitive  radio  (CR)  refers  to  the  emergence  of  technology 
that  combines  software  defined  radio  (SDR),  dynamic  spectrum  access  (DSA), 
networking,  and  artificial  intelligence  (AI)  techniques.  SDR  refers  to  radios  that  use 
software  to  define  some  or  all  physical  layer  functions  [1].  The  key  attribute  of  SDR  is 
that  through  software,  changes  can  be  rapidly  made  to  the  operating  characteristics  of  the 
radio.  Eor  example,  waveform  modulation  can  be  changed  by  issuing  a  software  command 
as  opposed  to  physically  changing  hardware  components.  The  capabilities  enabled  by  SDR 
lead  to  DSA.  DSA  is  the  ability  to  use  spectrum  that  is  available  in  time,  frequency,  or  space 
[2].  CR  capitalizes  on  both  SDR  and  DSA  by  enabling  intelligent  use  of  spectrum  through 
networks  and  devices  via  cognition.  Cognition  is  primarily  accomplished  through  the  use 
of  AI  to  make  decisions  in  support  of  user  policy. 

There  are  several  benefits  of  using  CR  that  solve  current  real-world  problems.  Eirst 
and  foremost,  CR  addresses  the  issue  of  spectrum  congestion  through  its  use  of  DSA. 
Spectrum  congestion  is  the  result  of  spectrum  being  allocated  by  a  central  governing 
authority;  some  frequency  bands  are  assigned  to  users  that  rarely,  if  ever,  use  their 
allocated  portion  of  the  spectrum  while  other  frequency  bands  are  over-utilized  by  many 
users.  Spectrum  congestion  has  become  a  more  visible  problem  in  recent  years  due  to  the 
explosive  growth  in  the  number  of  wireless  devices.  CR  helps  mitigate  spectrum  congestion 
by  intelligently  utilizing  unused  portions  of  the  spectrum.  Mitigating  spectrum  congestion 
improves  network  operation  and  improves  end-user  data  rates.  A  second  problem  CR 
addresses  is  tactical  in  nature.  CRs,  as  intelligent  agents,  can  utilize  their  knowledge  of 
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the  spectrum  to  self-organize,  covertly  communicate,  coordinate  action  against  the  enemy, 
and  provide  resiliency  and  redundancy  against  enemy  action.  Both  of  these  problems  that 
CR  address  are  stated  in  the  Air  Force  Technology  Horizons  report  [3]. 

1.1  Problem  Statement 

The  objective  of  this  research  is  to  develop  a  framework  by  which  CRs  can  be 
evaluated  as  a  complete  system.  Because  CR  spans  several  disciplines,  no  standardized  test 
methodology  has  yet  been  developed  for  use  in  test  and  evaluation  of  CRs  [4] .  Furthermore, 
because  CR  contains  AI,  the  long  term  behavior  and  performance  of  a  CR  may  be  unknown 
as  the  AI  evolves  in  reaction  to  its  environment,  making  test  and  evaluation  of  a  CR  more 
difficult.  According  to  Zhao,  “how  to  effectively  yet  trustfully  validate  a  CR  device  under 
varying  known  or  even  unknown  scenarios  is  an  open  issue  to  address”  [5]. 

However,  the  performance  and  behavior  of  a  CR  must  be  well  understood  if  CRs  are 
to  be  certified  and  used  outside  of  the  laboratory.  For  example.  Federal  Communications 
Commission  (FCC)  compliance  requires  devices  to  only  transmit  on  assigned  frequencies. 
Traditionally,  this  compliance  is  measured  through  hardware  verification  and  validation. 
However,  a  CR  may  have  the  hardware  capability  to  transmit  outside  of  assigned 
frequencies,  and  only  software  prevents  it  from  doing  so.  Ensuring  that  AI  software  meets 
the  same  level  of  compliance  as  traditional  hardware  verification  and  validation  is  an  area 
of  on-going  research.  As  a  result,  there  is  no  method  of  verification  and  validation  for 
CRs.  Methods  have  been  proposed  to  evaluate  certain  components  of  a  CR,  but  none  offer 
a  comprehensive  evaluation  of  a  complete  CR  system.  The  lack  of  a  framework  to  test 
CRs  as  a  complete  system  across  multiple  technology  domains  is  the  primary  problem  this 
research  seeks  to  address. 

Specifically,  the  goals  of  this  research  are  to: 

•  develop  a  test  methodology  to  evaluate  CRs  as  a  complete  system. 
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validate  the  developed  test  methodology, 


•  develop  a  CR  prototype  to  inelude  a  speetrum  sensor  for  use  in  testing  and 
development. 

Note  that  this  researeh  applies  only  to  physieal  CR  deviees,  not  to  theoretieal  or 
simulated  CR  deviees.  The  assumption  is  that  the  CR  under  test  is  a  real-world  device.  As 
a  real-world  device,  the  CR  can  be  seamlessly  moved  from  the  test  environment  to  being 
used  in  day  to  day  life.  Stated  another  way,  a  CR  is  considered  to  be  the  combination  of 
the  hardware  and  software  that  enables  cognitive  functionality,  not  simply  the  software. 
This  research  treats  the  words  CR,  cognitive  radio  system  (CRS),  and  cognitive  radio 
network  (CRN)  as  extensions  of  the  same  technology.  While  similar,  this  research  does 
not  directly  address  cognitive  network  (CN)s.  This  is  explained  further  in  Section  2.1. 

This  research  effort  partially  achieved  all  goals.  A  test  methodology  was  developed 
and  then  validated  and  shown  to  be  effective  in  evaluating  CRs  within  the  constraints  of 
this  research.  However,  the  scope  of  the  validation  does  not  justify  the  developed  test 
methodology  in  its  entirety.  A  working  CR  prototype  was  developed  for  use  in  testing. 
Also,  a  spectrum  sensor  was  developed  and  implemented  in  hardware  and  may  be  used  for 
future  research  efforts. 

1.2  Contributions 

This  research  provides  three  contributions  to  the  CR  field.  The  first  is  a  test 
methodology  that  can  be  used  to  standardize  CR  evaluations,  regardless  of  CR  architecture 
or  hardware- specific  implementation.  Secondly,  this  research  provides  a  new  way  of 
approaching  testing  cognition  by  measuring  device  behavior.  A  behavior-based  approach 
to  testing  and  evaluating  cognition  is  shown  to  be  effective  and  negates  the  need  to  evaluate 
cognition  on  a  component  level.  The  results  of  this  research  serve  as  a  “litmus  test”  in 
establishing  the  utility  of  behavior-based  testing.  Thirdly,  this  research  provides  a  CR 
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prototype  using  the  Wireless  Open  Aeeess  Research  Platform  (WARP)  board  from  Rice 
University.  The  CR  prototype  is  capable  of  sensing  the  environment,  reacting  to  the 
environment,  and  communicating  with  other  devices  using  multiple  waveforms.  These 
research  contributions  apply  not  only  to  CR  but  also  to  DSA  and  Opportunistic  Spectrum 
Access  (OSA)  fields  as  well. 

1.3  Overview 

Chapter  2  provides  a  detailed  background  on  existing  research  related  to  test  and 
evaluation  of  CRs.  The  chapter  begins  with  formal  definitions  of  what  a  CR  is  and 
then  examines  multiple  CR  architectures  that  have  been  proposed.  Following  that,  the 
chapter  presents  relevant  CR  testbeds  and  prototypes.  Finally,  the  chapter  examines  CR 
performance  metrics,  benchmarks,  and  methodological  aspects  that  pertain  to  development 
of  a  test  methodology. 

Chapter  3  presents  the  developed  test  methodology,  known  as  Cognitive  RAdio  Test 
Methodology  (CRATM).  CRATM  is  designed  to  address  the  issues  presented  in  Chapter  2. 
A  comparison  of  CRATM  with  the  literature  is  also  presented. 

Chapter  4  presents  the  methodology  to  evaluate  the  developed  test  methodology.  Due 
to  scope,  CRATM  is  not  evaluated  in  its  entirety.  Instead,  one  key  component  behind 
CRATM,  behavior-based  testing,  is  investigated.  Research  experiments  are  described  in 
this  chapter. 

Chapter  5  presents  the  results  of  the  experiments.  Empirical  results  are  collected  on 
an  implementation  of  CRATM  using  a  CR  prototype  in  regards  to  behavioral-based  testing. 

Chapter  6  presents  a  summary  of  the  experimental  results  and  of  the  developed  test 
methodology.  Conclusions  and  future  work  are  also  presented  in  Chapter  6. 
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II.  Related  Work 


The  CR  field  is  diverse  and  with  few  formal  systemic  categorizations  and  classifica¬ 
tions.  However,  the  boundaries  of  a  system  must  be  well  understood  if  it  is  to  be 
effectively  evaluated.  This  chapter  presents  a  current  assessment  of  the  CR  field  and  how  it 
is  classified.  The  various  definitions  of  CRs  are  examined  first,  followed  by  the  features  of 
an  assortment  of  CR  architectures.  After  that,  testbeds,  prototypes,  and  test  methodologies 
for  CRs  are  presented.  The  information  presented  in  this  chapter  is  intended  to  provide 
context  for  the  developed  test  methodology  in  Chapter  3. 

2.1  Cognitive  Radio  Definitions 

This  section  on  definitions  answers  “what  does  a  CR  do?”  as  opposed  to  the  following 
section  on  architectures,  which  answers  “how  does  a  CR  work?”.  In  this  section,  the  use 
of  the  term  ’’lower”  layers  refers  to  the  physical  layer  and  data  link  layers  while  “higher” 
layers  refers  to  the  network  through  application  layers. 

As  an  immature  field  of  research,  CR  has  varying  definitions  on  what  it  means  to  be  a 
CR.  The  lack  of  agreement  on  the  definition  of  a  cognitive  radio  has  direct  implications  on 
how  a  CR  is  tested.  A  methodology  that  does  not  test  the  intended  system  under  test  (SUT) 
will  not  be  very  useful. 

Prior  to  presenting  specific  definitions  on  CRs,  the  relationships  between  CRs,  CRSs, 
CRNs,  and  CNs  must  be  understood.  A  CR  is  a  single  device,  which  may  be  used  in  a 
CRN.  A  CRN  is  a  collection  of  cognitive  and  non-cognitive  devices  that  may  also  feature 
cognition  on  the  network  layers.  By  contrast,  a  CN  requires  cognition  at  the  network  level, 
but  does  not  require  the  use  of  CRs.  A  CRS  is  used  in  this  research  as  an  all-encompassing 
term  to  describe  either  a  CR  or  a  CRN.  The  test  methodology  developed  in  this  research 
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is  proposed  to  be  extensible  to  CNs;  however,  its  applieations  to  CNs  are  not  directly 
investigated.  Instead  this  research  investigates  only  CRs  and  CRNs  as  CRSs. 

The  remainder  of  this  section  details  historical  or  prominent  CR  definitions. 

2.1.1  Mitola,  1999. 

The  term  “cognitive  radio”  was  initially  coined  by  Joseph  Mitola  to  describe  wireless 
devices  and  networks  intelligent  enough  to  detect  user  communication  needs  and  provide 
wireless  services  in  response  to  those  needs  [6].  A  CR  is  a  SDR  that  makes  use  of  the 
radio  environment  intelligently  through  automated  reasoning  about  the  needs  of  the  user. 
In  Mitola’s  framework,  the  radio  environment  is  shared  and  represented  through  a  Radio 
Knowledge  Representation  Language  (RKRL).  The  RKRL  provides  a  standard  language 
through  which  all  device  knowledge  can  be  shared  with  other  devices,  including  knowledge 
of  the  spectrum,  local  policy,  network  information,  device  information,  and  user  needs. 
In  this  framework,  CRs  employ  a  cognition  cycle  based  on  the  Observe-Orient-Decide- 
Act  (OODA)  loop  [7],  but  with  the  addition  of  plan,  learn,  and  act  stages  such  that  it 
becomes  a  OOPDLA  cycle  [6]. 

It  is  important  to  point  out  that  Mitola  views  CR  from  the  perspective  of  a  singular 
CR  node  interacting  with  existing  networks  and  devices,  regardless  if  those  networks  and 
devices  are  cognitive  or  not.  The  primary  means  of  cognition  for  Mitola  is  the  use  of  the 
RKRL  in  conjunction  with  a  priori  knowledge  to  change  the  behavior  of  the  radio  at  low 
layer  levels  [6].  In  other  words,  CR  is  an  extension  of  SDR  in  that  it  intelligently  uses  the 
PHY/MAC  layers.  Higher  layer  cognition  is  not  expected. 

2.1.2  Kantor,  1999  and  Thomas,  2005. 

By  contrast,  a  fellow  doctoral  student  with  Mitola,  Theo  Kantor,  presented  the  concept 
of  a  cognitive  network  as  a  network  with  memory  at  the  same  time  that  Mitola  presented 
his  concept  of  CR  [8]  (interestingly,  both  shared  the  same  advisor).  The  idea  of  a  cognitive 
network  is  formalized  in  [9]  as  a  network  that  has  cognitive  processes  that  sense,  plan. 
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decide,  learn,  and  act  on  network  conditions  while  taking  into  account  end-to-end  goals. 
This  cognitive  network  acts  across  all  layers,  not  just  the  PHY/MAC  layer.  In  essence,  a 
CN  takes  the  properties  of  a  CR  and  implements  those  properties  across  all  layers.  The 
key  difference  between  a  CN  and  a  CR  is  that  the  CN  takes  into  account  overall  end-to-end 
goals  as  opposed  to  node-specific  user  objectives  [10].  CNs  may  or  may  not  include  low 
layer  level  cognition  such  as  CRs  [11]. 

2.1.3  Cognitive  Radio  Networks. 

From  the  foundations  of  CRs  and  CNs,  the  idea  of  a  CRN  has  emerged  in  which  each 
end-user  node  in  the  network  is  a  CR  [12].  In  this  model,  both  a  cognitive  network  and 
a  cognitive  radio  can  co-exist,  allowing  cross-layer  optimization  depending  on  the  level 
of  cognition  of  the  system.  This  model  allows  integration  of  node-specific  CR  techniques 
with  the  end-to-end  goals  of  CNs. 

2.1.4  Haykin,  2005. 

Haykin  in  [13]  presents  the  cognitive  radio  as  an  intelligent  wireless  communication 
system  that  is  aware  of  the  environment,  uses  understanding-by-building  to  learn  from  the 
environment,  and  adapts  to  RF  stimuli  with  the  objectives  of  highly  reliable  communication 
whenever  and  wherever  needed  and  to  efficiently  utilize  the  spectrum.  Haykin  envisions  a 
CR  as  adapting  its  operating  parameters,  such  as  transmit  power,  carrier  frequency,  and 
modulation  in  order  to  fulfill  these  objectives.  Haykin  presents  the  spectrum  hole  as  the 
means  in  which  a  CR  operates  around  existing  users.  A  spectrum  hole  is  a  frequency  that 
is  not  being  used  by  the  primary  user  (PU)  at  a  specific  time  and  geographic  location. 
Note  that  Haykin,  like  Mitola,  is  primarily  focused  on  improving  communications  at  the 
physical  layer  by  efficiently  using  the  spectrum.  Haykin  also  proposes  using  interference 
temperature  (discussed  further  in  Section  2.4.2)  as  a  performance  metric  and  radio  scene 
analysis  as  a  means  of  evaluating  detection  of  spectrum  holes. 
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2.1.5  IEEE  802.22  and  DySPAN. 

The  IEEE  has  long  been  working  towards  developing  a  standard  for  a  real-world  CR. 
There  has  been  two  parallel  tracks-one  that  is  developing  generie  supporting  standards  and 
another  developing  a  speeifie  implementation  ealled  IEEE  802.22.  The  first  eommittee  to 
work  towards  developing  supporting  standards  was  the  IEEE  1900  standards  eommittee. 
This  eommittee  evolved  into  the  Standards  Coordinating  Committee  41  (SCC41),  which  in 
turn  evolved  into  the  Dynamic  Spectrum  Access  Networks  (DySPAN)  committee  [14,  15]. 
DySPAN’s  working  definition  of  a  CR  is  a  “radio  in  which  communication  systems  are 
aware  of  their  environment,  internal  state,  and  location  and  can  make  decisions  about  their 
radio  operating  behavior  based  on  that  information”  and  “utilizes  software  defined  radio, 
adaptive  radio,  and  other  technologies  to  autonomously  adjust  its  behavior  or  operations  to 
achieve  the  desired  objectives”  [16]. 

The  IEEE  802.22  standard  [17-19]  is  working  towards  a  CR  implementation  that 
follows  the  standards  presented  by  DySPAN.  The  IEEE  802.22  standard  proposes  to 
utilize  unused  portions  of  the  television  spectrum  without  causing  interference  to  existing 
television  receivers.  A  spectrum  broker,  or  spectrum  manager,  is  inherent  to  the  operation 
of  IEEE  802.22  [19].  A  spectrum  broker  is  a  device  responsible  for  dynamic  assignment 
of  channels  to  secondary  devices.  IEEE  802.22  assumes  that  there  are  base  stations  which 
facilitate  spectrum  coordination  by  acting  as  spectrum  brokers.  Individual  sensing  nodes 
may  feed  local  spectrum  information  to  the  base  stations  [17].  The  base  station  spectrum 
broker  has  a  geolocation  database  of  known  licensed  transmitters,  but  supplements 
spectrum  knowledge  by  using  localized  spectrum  sensing  [19]. 

Both  the  IEEE  802.22  and  DySPAN  standards  do  not  actually  require  (nor  currently 
have  standards  for)  cognition  at  the  CR  level.  Instead,  each  node  on  the  network  only 
needs  to  implement  the  policies  dictated  by  the  spectrum  broker  [15,  20].  Sensing  nodes 
are  required  to  feed  spectrum  knowledge  back  to  the  central  base  station,  though  this  act 
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does  not  require  eognition.  Instead,  all  eognition  oeeurs  via  a  cognitive  engine  ( CE)  loeated 
at  the  base  station.  Non-intelligent  deviees  eonsult  a  speetrum  database,  whieh  is  ereated 
and  maintained  by  the  intelligent  CE  [19,  20]. 

DySPAN  also  presents  two  models  of  speetrum  alloeation  and  usage:  owned  and 
eommon.  In  the  owned  model,  sole  ownership  of  the  speetrum  rests  with  a  user  or  ageney 
that  has  bought  that  portion  of  speetrum.  In  the  common  model,  the  speetrum  is  available 
to  all  users  where  the  users  agree  upon  an  etiquette  to  operate  by  [16]. 

2.1.6  Federal  Communications  Commission. 

The  FCC  has  a  loose  definition  of  eognitive  radio  that  is  better  deseribed  as  a  definition 
of  software  defined  radio.  At  the  present,  the  FCC  does  not  address  eognitive  radio  apart 
from  SDR  as  it  does  not  believe  that  the  fundamental  nature  of  speetrum  poliey  is  ready 
to  ehange  [21].  The  FCC  defines  a  CR  (or  SDR)  as  “a  radio  that  ineludes  a  transmitter  in 
whieh  the  operating  parameters  of  frequeney  range,  modulation  type  or  maximum  output 
power...  ean  be  altered  by  making  a  ehange  in  software  without  making  any  ehanges  to 
hardware  eomponents  that  affeet  the  radio  frequeney  emissions.”  [21]  The  FCC  proposed 
using  interference  temperature  as  a  metrie  for  evaluating  the  eo-existenee  of  unlieensed 
users  with  lieensed  users  [22]. 

2.1. 7  Wireless  Innovation  Forum. 

The  Wireless  Innovation  Forum,  formerly  known  as  SDRForum,  presents  a  eognitive 
radio  as  simply  a  radio  that  is  “eapable  of  making  deeisions  and  seleeting  or  modifying 
the  operating  parameters  of  a  radio”  [1].  The  SDRForum  views  a  CR  as  a  SDR  that  is 
eontrollable  at  the  physieal  layer  while  all  other  layers,  exeept  applieation,  are  part  of 
the  eognition  proeess.  The  goal  of  the  SDRForum  framework  is  to  make  the  eognition 
operation  transparent  to  higher  layers  [1]. 
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2.1.8  DARPA  XG  Program. 

Completed  in  2006,  the  next-generation  (XG)  program  used  DSA  teehniques  to 
operate  without  eausing  interferenee  to  pre-existing  non-eooperative  users  [23].  The  XG 
program  does  not  elaim  to  use  a  CR  per  se,  but  as  an  aetual  implementation  of  a  DSA 
deviee,  utilizes  many  CR  teehniques  that  are  part  of  a  basie  CR.  The  three  XG  sueeess 
eriteria  of  relevanee  to  CR  applieations  are: 

•  Not  eause  harm  to  existing  users 

•  Form  and  maintain  eonneeted  networks 

•  Add  value  by  effieiently  using  the  speetrum 

Taken  together,  these  three  eriteria  form  a  definition  on  what  a  CR  does  [23]. 

2.1.9  Cognitive  Radio  Definitions  Summary. 

In  general,  these  definitions  agree  that  CRSs  should  be  able  to  both  sense  the 
environment  and  autonomously  adapt  to  ehanging  eonditions  but  differ  as  to  to  the  depth 
of  eognitive  funetionality,  situation  awareness,  and  where  eognition  takes  plaee  [24].  A 
summary  of  eognitive  radio  definitions  as  presented  in  this  seetion  is  shown  in  Table  2.1. 

2.2  Architectures 

The  arehiteeture  of  a  CR  direetly  affeets  how  it  is  implemented,  and  therefore 
tested.  This  seetion  presents  an  overview  of  key  CR  arehiteetural  features,  as  well  as  two 
representative  CR  arehiteetures.  In  general,  a  CR  requires  a  means  of  sensing  the  speetrum, 
sharing  speetrum  knowledge,  eombining  speetrum  knowledge  with  the  knowledge  of  other 
deviees,  establishing  speetrum  eooperation  with  other  deviees,  and  eommunieating  with 
the  network.  Implieit  in  these  aetivities  is  a  CE,  whieh  forms  the  nueleus  of  the  AI.  The 
CE  implements  user  and  external  poliey  to  aeeomplish  user  goals.  Another  implieit  feature 
is  network  topology. 
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Table  2.1:  Different  Capabilities  On  What  Is  Neeessary  To  Be  Called  A  Cognitive  Radio 
Adapted  from  [24] 
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All  CR  arehiteetures  are  governed  by  the  OODA  loop.  A  representative  version  of  the 
OODA  loop  as  it  pertains  to  CR  is  shown  in  Figure  2.1'.  This  eognition  eyele  forms  the 
simplest  possible  framework  for  understanding  the  reaetion  of  the  CR  to  the  environment. 
Note  that  though  it  is  ealled  a  eyele,  stages  ean  oeeur  eoneurrently.  For  instanee,  the  observe 
stage  may  oeeur  while  the  act  stage  is  underway. 

In  CRs,  the  observe  stage  eorresponds  to  speetrum  sensing  and  speetrum  eooperation. 
The  orient  stage  eorresponds  to  applying  user  poliey  based  on  observations.  The  decision 
stage  eorresponds  to  ehoosing  a  eourse  of  aetion  to  adapt  to  the  observed  environment,  and 
'Reproduced  from  [25] 
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Figure  2.1:  Cognitive  Radio  Cognition  Cyele  [26] 


the  act  stage  eorresponds  to  the  implementation  of  the  deeision  and  performing  normal 
operations. 

In  terms  of  CR,  the  OODA  loop  ean  be  illustrated  as  follows.  The  speetrum  is  broken 
up  into  10  ehannels.  While  eurrently  operating  on  ehannel  2,  a  CR  senses  the  speetrum  and 
deteets  a  PU  on  ehannel  3.  This  information  is  shared  with  other  CR  deviees  on  the  network 
(observe).  A  master  node  reeeives  this  speetrum  information  and  eombines  the  speetrum 
information  with  its  speetrum  database  (orient).  The  master  node  notiees  that  the  PU  on 
ehannel  3  will  be  interfered  with  by  several  nodes  based  on  their  relative  proximity.  The 
master  node  deeides  that  eommunieation  is  best  served  by  ehanging  to  ehannel  6  (decide). 
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The  master  node  eommands  the  CRs  to  jump  to  this  new  ehannel  and  to  eontinue  operation 
{act). 

2.2.1  Spectrum  Sensing,  Sharing,  and  Cooperation. 

The  issue  of  speetrum  sensing,  sharing,  and  eooperation  is  well  established  in  the 
literature.  Readers  are  referred  to  [2, 12,  27-29]  for  a  eomprehensive  treatment  of  speetrum 
sensing  and  eooperation.  For  the  purposes  of  developing  a  test  methodology,  it  suffiees 
to  state  that  there  are  many  means  to  ehoose  from  to  deteet  PUs  and  form  speetrum 
knowledge.  Speetrum  sensing  algorithms  range  from  simple  algorithms  that  only  look 
for  inereased  power  relative  to  noise  floor  to  algorithms  that  deteet  features  beneath  the 
noise  floor.  Eaeh  speetrum  sensing  algorithm  offers  a  different  degree  of  resolution  with 
eorresponding  eomputational  eomplexity. 

Two  general  means  of  reeording  and  sharing  speetrum  knowledge  have  been  proposed. 
The  first  is  to  utilize  a  language-based  representation  sueh  as  the  RKRL  proposed  by  Mitola 
[6].  The  seeond  is  to  utilize  databases.  A  prominent  database  form  of  speetrum  knowledge 
is  the  radio  environment  map  (REM).  In  the  REM,  database  elements  inelude  geographieal 
information,  serviees  and  networks,  regulations  and  poliey,  the  aetivity  profile  of  radio 
deviees,  and  learned  experienee  [30].  If  a  CE  is  the  brain,  the  REM  is  the  memory  the 
brain  draws  upon.  The  IEEE  802.22  /  DySPAN  standards  eombine  elements  from  both 
language-based  representations  and  database  forms  of  speetrum  knowledge  [19,  20]. 

Speetrum  information  sharing  and  cooperation  is  closely  related  to  the  CR  topology 
presented  in  Section  2.2.2. 

2.2.2  Topology. 

There  are  two  primary  CR  topologies,  centralized  and  distributed  [12,  28].  In  the 
centralized  topology,  a  central  node  acting  as  a  base  station  or  access  point  serves  to 
facilitate  communication  with  secondary  user  (SU)s.  In  general,  the  base  station  controls  all 
of  the  SUs  within  transmission  range.  SUs  feed  spectrum  sensing  data  forward  to  the  base 
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station.  The  centralized  topology  may  or  may  not  require  two  communication  channels- 
one  for  the  primary  communication/observation  channel  and  one  for  reporting  spectrum 
information  to  the  base  station. 

The  second  topology  is  the  distributed  topology.  This  topology  can  be  thought  of 
as  an  ad  hoc  or  mesh  network.  In  this  topology,  SUs  communicate  directly  with  each 
other  without  any  central  node.  Spectrum  data  is  shared  between  SUs  and  each  SU  can 
independently  make  a  decision  on  the  appropriate  communication  protocol.  Optionally, 
one  node  can  be  designated  as  a  head  node  with  spectrum  decision  authority.  As  with  the 
centralized  topology,  a  distributed  topology  may  or  may  not  require  two  communication 
channels. 

2.2.3  Cognitive  Engine. 

The  CE  is  the  intelligence  behind  a  CR  [31].  Research  into  CEs  overlaps  with 
AI  research.  Technologies  behind  CEs  include  genetic  algorithms,  case-based  reasoning 
systems,  expert  systems,  or  combination  of  these  approaches  [24,  31].  According  to  [5], 
there  are  two  approaches  to  implementing  a  CE  within  a  CR.  The  first  is  a  low-complexity 
CE  approach.  In  this  approach,  CRs  use  REMs  or  similar  database  both  locally  and 
globally  for  situational  awareness,  efficient  learning,  and  fast  adaptation.  The  goal  with 
this  approach  is  fast  adaptation  for  low  cost  and  complexity.  The  second  approach  is  to 
use  high  performance  computing.  In  this  approach,  CRs  utilize  either  multi-core  CPUs 
or  offload  data  computations  to  the  cloud.  Both  of  these  approaches  assume  a  centralized 
topology. 

When  developing  a  test  methodology,  it  is  important  to  have  a  framework  that  is  not 
limited  to  testing  only  one  permutation  of  the  CE,  but  is  all-inclusive. 

2.2.4  Miscellaneous  Features. 

Other  features  of  a  CR  architecture  are  derived  from  real-world  technology  constraints. 
Eor  instance,  a  working  CRS  may  need  a  common  control  channel  other  than  the 
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primary  communications  channel.  This  common  control  channel  serves  as  a  means  to 
share  speetrum  information  or  issue  eommands  to  nodes  in  the  event  that  the  primary 
eommunieations  ehannel  is  not  useable.  The  eommon  eontrol  ehannel  is  usually  thought 
of  as  a  low  bandwidth  alternative.  Another  arehiteetural  feature  is  whether  or  not  the  CRS 
is  homogeneous  or  heterogeneous.  These  features  are  often  implieitly  assumed  in  some 
CR  definitions  or  arehiteetures.  The  test  methodology  developed  in  this  researeh  seeks  to 
avoid  tying  itself  to  any  partieular  CR  definition  and  instead  aims  to  be  flexible  enough  to 
aeeommodate  all  definitions  and  arehiteetures. 

2.2.5  IEEE  802.22  Architecture. 

The  IEEE  802.22  proposal  serves  as  one  type  of  CR  arehiteeture.  The  IEEE  802.22 
standard  [17,  19]  is  intended  to  allow  SU  deviees  to  eo-exist  with  television  stations  PUs  by 
using  white  spaees  (i.e.  speetrum  holes).  IEEE  802.22  is  a  wireless  regional  area  network 
(WRAN).  The  ehief  problem  is  ensuring  that  various  SU  deviees  do  not  interfere  with  PU 
reeeivers.  To  do  this,  IEEE  802.22  uses  a  speetrum  broker  loeated  at  the  base  station.  The 
speetrum  broker  polls  CR  nodes  for  speetrum  information  and  eompiles  that  information 
into  a  global  REM  database.  Then,  the  speetrum  broker  alloeates  speetrum  to  eaeh  node 
based  on  the  global  REM  database  and  the  poliey  goals  of  the  network.  In  the  IEEE 
802.22  arehiteeture,  nodes  do  not  have  to  be  eognitive  so  long  as  they  ean  implement  the 
instruetions  of  the  speetrum  broker.  However,  there  is  a  requirement  for  speetrum  sensing 
to  oeeur  away  from  the  base  station  at  the  nodes. 

2.2.6  AFIT  Cognitive  Radio  Lab  Architecture. 

The  AEIT  Cognitive  Radio  Eaboratory  (ACRO),  based  on  work  by  MeEean  [32], 
puts  forth  an  ad-hoe  CR  arehiteeture.  In  the  ACRO  arehiteeture,  the  nodes  follow  the 
seven  step  proeess  shown  in  Eigure  2.2.  In  this  framework,  individual  nodes  sense  the 
speetrum  and  share  REMs  with  eaeh  other  using  a  protoeol  sueh  as  multieast.  The  REM 
in  this  arehiteeture  is  simply  a  binary  veetor,  where  eaeh  position  in  the  veetor  is  a  1  or  0 
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Figure  2.2:  AFIT  Cognitive  Radio  Lab  Arehiteeture  System  Funetional  Diagram  [32] 


depending  on  if  the  eorresponding  ehannel  is  oeeupied  or  not.  Implieit  in  the  arehiteeture 
is  a  master  node  that  eompiles  distributed  REMs  and  redistributes  a  shared  REM  baek  to 
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the  rest  of  the  nodes.  The  architeeture  does  not  speeify  if  the  REM  sharing  occurs  on  the 
primary  channel  or  on  a  common  control  channel.  The  architecture  uses  frequency  hopping 
when  selecting  waveforms. 

2.3  Testbeds 

Testbeds  are  a  critical  tool  in  the  evolution  of  CR  technology  from  theoretical  analysis 
to  real-world  hardware  that  will  be  used  outside  of  the  lab  [26].  Several  have  been 
developed  for  use  in  prototyping  CRs.  These  testbeds  generally  combine  real  hardware 
devices  with  emulation,  in  which  the  real  hardware  devices  can  communicate  through  a 
controlled  wireless  or  wired  physical  layer.  However,  some  testbeds  feature  no  emulation 
and  offer  no  direct  control  of  the  radio  frequency  (RF)  environment.  According  to 
[33],  every  CR  testbed  should  provide  the  ability  to  support  multiple  radios  (both  PUs 
and  SUs)  that  are  fully  controllable,  the  ability  to  connect  various  different  front-ends 
over  different  frequency  ranges,  the  ability  for  physical/link  layer  adaptation  and  fast 
information  exchange,  and  support  of  rapid  prototyping.  Problems  with  current  testbeds 
include  a  lack  of  large  scale  capabilities,  a  tendency  to  focus  on  802.1 1  networks,  limited 
integration  of  hardware,  and  systems  that  are  either  too  complex  or  too  simple  [34]. 

This  section  presents  two  approaches  to  building  and  operating  CR  testbeds.  Note 
that  the  specific  testbeds  described  in  this  section  are  predicated  on  testing  a  physical  radio 
device.  Systems  that  simulate  or  emulate  the  system  under  test  are  not  considered.  Also, 
systems  that  playback  traces  of  the  environment,  such  as  in  [35],  are  not  considered  in 
this  research.  Trace  based  testing  is  excellent  for  running  realistic  simulations  but  does 
not  lend  itself  towards  the  testing  of  physical  devices.  Furthermore,  testbeds  that  do  not 
enable  testing  of  unique  hardware  are  not  considered  in  this  research.  The  majority  of 
testbeds  in  the  literature  are  of  this  type-these  testbeds  have  a  fixed  hardware  RF  front-end 
and  are  focused  on  algorithm  development,  and  cannot  be  used  to  test  generic  hardware 
configurations. 
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The  testbeds  presented  in  this  seetion  are  also  predieated  on  offering  some  eontrol 
over  the  RF  environment.  Aecording  to  [36],  “researeh  aimed  at  evaluating  and  improving 
wireless  network  protoeols  and  applications  is  hindered  by  the  inability  to  perform 
repeatable  and  realistic  experiments.”  If  experiments  are  to  be  repeatable,  control  over  the 
RF  environment  is  necessary.  However,  this  must  be  balanced  by  the  requirement  for  a 
realistic  RF  environment.  The  ideal  wireless  experimentation  device  would  offer  repeatable 
results,  controllability,  cross-layer  realism,  run  real  experiments,  be  quickly  configurable, 
and  test  a  large  number  of  nodes.  Emulation  offers  the  easiest  way  to  meet  these  criteria. 

2.3.1  Emulation-based  Testbeds. 

Emulation  offers  a  means  of  tightly  controlling  the  variables  in  the  experiment  while 
still  offering  a  well-defined  dynamic  environment  which  can  be  simultaneously  realistic, 
fully  controllable,  repeatable,  and  diverse  [37-39].  Emulation  also  enables  the  outputs  of 
one  sub-system  to  be  fed  into  another  sub-system.  According  to  Borries  et  ah,  emulation 
is  particularly  well  suited  to  device  and  link  characterization  for  controlled  studies  in  the 
early  stages  of  a  project  into  physical  layer  effects  [39].  They  also  claim  that  emulator 
capabilities  are  unique  in  supporting  experiments  that  are  simultaneously  realistic,  fully 
controllable,  repeatable,  and  diverse. 

To  emulate  the  RE  environment,  there  are  three  main  approaches  [40].  The  first 
approach  is  to  scale  the  radio  signals  between  wireless  devices  via  attenuation.  The  second 
approach  is  to  emulate  just  the  channel  path  between  wireless  devices  using  a  hardware- 
based  channel  emulator.  The  third  approach  is  to  emulate  the  wireless  device  and  channel 
effects.  The  goal  of  this  research  is  to  investigate  how  real  wireless  devices  behave  in 
a  controlled  environment;  therefore,  the  use  of  a  hardware-based  channel  emulator  (the 
second  approach)  is  most  appropriate.  Existing  hardware-based  channel  emulator  testbeds 
include  the  CMU  Emulator,  WHYNET,  BEE2,  and  the  DYSE.  Alternative  testbeds,  ORBIT 
and  MINT,  utilizes  the  first  approach. 
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2.3.1. 1  CMU  Emulator. 


The  Carnegie  Mellon  University  (CMU)  Emulator  [39]  is  a  15  node  emulator  that 
has  been  in  use  since  2007.  The  emulator  supports  the  full  2.4  GHz  ISM  band.  The 
wireless  nodes  are  laptops  with  802.11b  interfaces,  though  the  system  has  the  capability 
to  incorporate  other  RF  devices.  A  Linux  server  acts  as  the  environment  controller  and 
controls  the  channel  parameters  that  are  computed  by  field  programmable  gate  array 
(FPGA).  The  output  of  the  wireless  nodes  is  fed  to  FPGAs  for  channel  path  processing. 
The  FPGAs  emulate  the  desired  signal  effects  such  as  attenuation,  fading,  multi-path,  and 
interference.  The  FPGAs  also  combine  the  processed  signals  into  an  output  signal,  after 
which  it  is  converted  to  RF  and  streamed  back  to  the  wireless  nodes. 

2.3.1.2  WHYNET. 

The  Wireless  Hybrid  Network  [40],  WHYNET,  is  a  framework  that  enables  integration 
of  physical,  simulation,  and  emulation  components.  The  system  is  built  to  understand 
cross-layer  protocol  interactions  by  exploiting  physical  layer  flexibility.  The  system  is 
designed  to  provide  a  realistic,  scalable,  flexible,  and  cost-effective  evaluation  environment. 
The  physical  components  of  WHYNET  consist  of  wireless  devices  that  communicate  over 
real  wireless  channels.  The  emulated  components  consist  of  a  hardware-based  channel 
emulator  and  a  wireless  network  emulator.  The  hardware-based  channel  emulator  computes 
the  emulated  signal  between  wireless  nodes.  The  wireless  network  emulator  emulates 
the  devices  and  combines  both  device  and  channel  path  behavior  into  software  models. 
WHYNET  also  has  simulation  capabilities.  The  testbed  infrastructure  includes  a  variety  of 
communications  testbeds  and  devices. 

2.3.1.3  BEE2. 

The  Berkeley  Emulation  Engine  2  (BEE2)  [33]  is  a  an  FPGA-based  channel  emulator 
that  supports  up  to  18  wireless  nodes.  The  BEE2  designers  envisioned  the  platform  to  serve 
as  a  testing  ground  for  CRs,  enabling  both  PUs  and  SUs  to  operate  in  the  same  space  for 
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testing.  The  PUs  are  direetly  eontrollable  and  preeise  interferenee  measurements  ean  be 
made.  The  BEE2  unit  eontains  five  Virtex-2  Pro  70  EPGAs,  of  whieh  four  are  used  for 
ehannel  emulation  and  one  is  used  for  eontrol.  The  BEE2  aeeepts  up  to  20  MHz  of  RE 
bandwidth  and  ean  operate  within  the  entire  2.4  GHz  ISM  band. 

2.3. L4  ORBIT  and  MiNT. 

The  Open  Aeeess  Researeh  Testbed  for  Next-Generation  Wireless  Networks  (ORBIT) 
is  a  two-tier  laboratory  emulator  and  field  trial  network  testbed  designed  to  aehieve 
reprodueibility  of  experimentation  while  also  supporting  real-world  settings.  The  testbed 
has  over  400  802.1 1  a/b/g  nodes  laid  out  on  20  by  20  grid.  Researeh  at  ORBIT  is  foeused 
on  higher- layer  network  protoeols  with  limited  physieal  layer  researeh  [41,  42].  ORBIT 
emulates  various  environments  by  attenuating  and  mixing  signals.  The  Miniaturized 
Mobile  Multi-Hop  Wireless  Network  Testbed  (MiNT)  [43]  is  a  similar  testbed  to  ORBIT, 
though  it  foeuses  on  802.11b  networks. 

2.3. L5  CREW. 

Cognitive  Radio  Experimentation  World  (CREW)  Eederated  Testbed  [37]  eonsists  of 
five  geographieally  separated  laboratory  testbeds  loeated  throughout  Europe.  The  CREW 
testbeds  are  intended  to  be  able  to  perform  experimentally  driven  researeh  aeross  all  layers 
and  to  eapitalize  on  the  strengths  of  individual  testbeds  for  different  applieations.  CREW 
is  foeused  on  eognitive  networks  and  dynamie  speetrum  aeeess  (DSA)  researeh.  The 
CREW  approaeh  uses  traees  reeorded  in  one  testbed  to  play  baek  in  another  to  enable  a 
form  of  emulation.  The  ehief  limitation  of  this  approaeh  is  that  diserete  testbeds  will  not 
eause  RE  interferenee  with  eaeh  other,  whieh  means  that  the  eognitive  system  is  not  fully 
eharaeterized. 

2.3. 1.6  DYSE. 

Dynamie  Speetrum  Environment  Emulator  (DYSE)  [44]  is  a  maehine  that  tests 
wireless  deviees  by  emulating  ehannel  path  eonditions  in  near  real-time  for  both  real  and 
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emulated  wireless  deviees.  For  eaeh  node,  controllable  parameters  include  movement, 
delay,  and  antenna  gain  patterns.  The  DYSE  has  predefined  channel  settings  that  include 
air-to-air,  air-to-ground,  rural,  suburban,  urban,  and  dense  urban.  The  DYSE  offers  control 
of  channel  parameters  for  path  loss,  fading,  and  multipath.  Real  world  terrain  data  is  fed 
into  the  system  via  the  Google  Static  Maps  API  service  or  through  Terrain  Integrated  Rough 
Earth  Model  (TIREM)  data.  Therefore,  wireless  paths  between  nodes  can  be  calculated  for 
a  wide  variety  of  realistic  test  conditions,  making  it  well  suited  for  diverse  experimentation 
with  DSA  radios. 

The  DYSE  operator  specifies  the  location  of  each  node  and  whether  it  is  a  transmitter 
or  receiver  via  a  GUI.  The  operator  also  specifies  the  antenna  gain  patterns  and  movement 
of  each  node  through  the  GUI  as  well.  The  channel  path  characteristics  are  automatically 
populated  using  the  altitude  and  location  of  each  node  from  the  TIREM  dataset  unless  the 
operator  chooses  to  specifically  define  the  channel  path  characteristics. 

The  DYSE  can  operate  with  all  virtual  nodes  (all  nodes  are  emulated),  with  only 
physical  nodes  attached,  or  with  a  combination  of  both  physical  and  virtual  nodes  (mixed 
mode).  In  virtual-only  mode,  the  DYSE  can  emulate  up  to  100  nodes  at  once.  In  this  mode, 
the  system  operates  in  snapshots  since  real-time  operation  is  not  necessary.  By  contrast, 
the  physical-only  mode  requires  and  operates  with  full  real-time  streaming.  The  gross¬ 
processing  delay  is  non-time  varying  but  is  fixed  for  a  specific  scenario.  Eor  a  nominal  4 
physical  unit  scenario  (16  channels),  the  gross  processing  delay  is  approximately  500  //s. 
The  mixed  virtual  and  physical  mode  is  the  most  complex.  Real-time  operation  is  capable 
for  simulations  with  low  complexity;  however,  at  a  certain  point  the  system  can  no  longer 
support  real-time  operation  and  continuity  of  signals  cannot  be  guaranteed.  This  crossover 
threshold  is  scenario  dependent.  Two  major  factors  affecting  real-time  operation  in  mixed 
mode  are  the  number  of  virtual  nodes  and  the  complexity  of  virtual  nodes. 
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The  DYSE  hardware  eonsists  of  an  workstation  eomputer  hosting  a  graphies 
proeessing  unit  (GPU)  and  an  RF  interface  tray  for  interfacing  with  physical  devices.  The 
workstation  utilizes  a  single  6-core,  3.33  MHz  Intel  i7-980x  CPU  with  12GB  of  triple 
channel  DDR3  DRAM.  The  workstation  runs  on  Linux  Ubuntu.  The  GPU  utilizes  the 
Nvidia®CUDA  API.  The  GPU  is  responsible  for  signal  propagation  computation.  The 
RF  interface  tray  consists  of  four  Universal  Software  Radio  Peripheral  (USRP)  SDRs 
from  Ettus  Research.  Each  USRP  performs  digital-down/up  conversion  between  RF  and 
baseband.  The  RF  interface  tray  is  limited  to  12.5  MHz  of  bandwidth  per  USRP.  The 
USRPs  operate  over  the  frequency  range  from  50  to  2200  MHz. 

2.3.2  Non-Emulation  Testbeds. 

There  are  a  large  number  of  non-emulation  testbeds  used  by  the  CR  community.  Since 
this  research  is  focused  on  testbeds  that  offer  control  of  the  physical  layer,  they  are  not 
presented  here.  The  reader  is  referred  to  [5,  34,  37,  39,  42,  45]  for  information  on  non¬ 
emulation  testbeds. 

2.4  Performance  Metrics 

Performance  metrics  for  CR  may  measure  the  performance  of  a  CR  component,  an 
individual  CR  node,  or  the  CR  system  as  a  whole.  Some  performance  metrics,  such 
as  those  that  apply  to  spectrum  sensing,  may  apply  across  these  domains.  The  use  of 
uniform  performance  metrics  is  necessary  for  standardized  evaluation.  The  most  common 
performance  metrics  are  presented  here;  however,  the  authors  of  [24]  provide  an  exhaustive 
treatment  of  performance  metrics  for  CRs. 

2.4.1  Spectrum  Sensing. 

Research  on  spectrum  sensing  has  focused  on  the  ability  to  detect  a  PU.  The  following 
are  established  performance  metrics. 
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•  Probability  of  Primary  User  Detection:  This  metric  is  the  probability  that  the  PU 
is  properly  detected.  This  detection  has  both  a  false  negative  and  a  false  positive 
component.  The  false  negative  component  is  the  probability  of  not  detecting  the  PU, 
while  the  false  positive  component  is  the  probability  that  a  PU  is  detected  that  is  not 
present  [33].  The  false  positive  is  also  referred  to  as  the  probability  of  false  alarm. 
Mathematically  [26],  the  null  hypothesis  is  Hq,  or  the  hypothesis  that  there  is  no  PU 
on  the  channel.  The  test  hypothesis  is  Hi,  or  that  there  is  a  PU  on  the  channel.  The 
test  statistic  from  the  spectrum  sensing  unit  is  T,  and  is  compared  to  a  predetermined 
threshold  A.  The  probability  of  false  alarm,  Pp,  denotes  the  probability  that  the 
hypothesis  test  chooses  Hi  while  it  is  actually  Hq.  The  probability  of  detection,  Pd, 
denotes  the  probability  that  the  test  correctly  decides  Hi . 

Pf  =  P{T>A\Ho)  (2.1) 

Pd  =  P(.T>A\Hi)  (2.2) 

The  probability  of  PU  detection  is  best  assessed  at  the  component  level,  unless 
absolute  spectrum  knowledge  of  both  the  SUT  and  the  environment  is  known. 

•  Time  to  Detect  a  Primary  User:  T detect  is  the  time  between  the  initial  PU  signal 
transmission  and  the  time  the  CRS  is  updated  with  correct  spectrum  knowledge  [33]: 

Tdetect  =  Trem  -  Tpujx  (2.3) 

As  with  the  previous  metric,  the  time  to  detect  a  PU  is  difficult  to  measure  in  hardware 
and  is  best  suited  for  laboratory  or  simulating  testing. 

2.4.2  Avoiding  Causing  Interference. 

The  following  metrics  pertain  to  the  ability  of  a  CR  to  avoid  causing  harmful 
interference  to  existing  users.  According  to  Kolodzy,  “the  design  and  operation  of 
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RF  equipment  including  communications  and  emitting  non-communications  devices  are 
predicated  upon  preventing  and/or  mitigating  electronic  interference”  [46].  In  accordance 
with  this  statement,  this  research  assumes  that  the  ability  to  avoid  causing  interference  is  a 
mandatory  capability  of  CR. 

•  Time  to  Evacuate  a  Channel:  Tevac  is  the  time  it  takes  a  CR,  or  CRS,  to  detect 
a  signal  {T detect)  plus  the  time  it  takes  for  the  node  to  process  the  spectrum 
knowledge,  decide  to  cease  transmitting,  and  to  ultimately  cease  transmitting 
{T cease)  [23,33]: 

Tevac  =  T detect  +  T cease  (2.4) 

This  metric  was  used  in  the  XG  program,  where  success  was  defined  as  a  channel 
abandonment  time  of  less  than  500  ms  [23]. 

•  Interference  Temperature:  Interference  temperature  was  first  proposed  by  the  FCC  as 
a  way  to  set  a  limit  on  how  much  interference  an  unlicensed  user  causes  to  licensed 
users  [22] .  Interference  temperature  is  a  measure  of  how  much  RF  power  is  available 
at  a  receiving  antenna  to  be  delivered  to  a  receiver  that  incorporates  the  power 
generated  by  other  emitters  and  noise  sources  [46-48].  The  equation  for  interference 
temperature  is: 


Ti{fc,B)  =  ^^^^  (2.5) 

kB 

In  this  equation,  Pj  (/c,  B)  is  the  average  interference  power  in  Watts  for  a  center 
frequency  fc  and  bandwidth  B  measured  in  Hertz.  Parameter  k  is  Boltzmann’s 
constant.  Interference  temperature  is  similar  to  noise  temperature  [47]. 

Interference  temperature  was  developed  to  address  CRs  utilizing  the  underlay 
paradigm  (see  Section  2.8)  [22].  However,  it  is  suitable  for  use  in  other  paradigms  as 
well. 
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•  Probability  of  Collision:  Instead  of  measuring  interferenee  temperature,  interferenee 
ean  be  measured  as  the  likelihood  that  the  SU  transmits  simultaneously  with  the  PU 
[2].  Due  to  the  binary  results  of  this  metrie,  it  does  not  offer  fidelity  or  adequate 
resolution  for  assessing  the  impaet  of  the  SU  on  the  PU.  However,  it  is  simple  to 
measure  in  a  hardware  SUT  deviee. 

2.4.3  Cognitive  Radio  Artificial  Intelligence. 

For  CRSs,  Zhao  et  al  [5]  propose  to  use  “IQ”  and  “EQ”  to  test  CRs,  where 
the  IQ  is  the  intelligenee  of  a  single  CR  node  while  the  EQ  is  the  intelligenee  of  a 
eross-node/network  eollaboration  of  the  CRS.  EQ  is  also  the  eapability  of  obtaining 
global  environmental  awareness  through  eollaborative  sensing  with  other  nodes  or  through 
network  infrastrueture.  CR  AI  performanee  may  also  be  viewed  as  a  set  of  tasks  to  be 
performed,  and  evaluated  as  sueh  [13]. 

2.4.4  Communication  Performance. 

Legaey  performanee  metries  for  evaluating  eommunieation  performanee  are  suffieient 
for  CRS  applieations.  Legaey  performanee  metries  inelude  measuring  bit  error  rate  (BER), 
paeket  error  rate  (PER),  throughput,  or  network  Quality  of  Serviee  (QoS)  for  the  CRS. 
These  metries  are  well  established  and  not  diseussed  further  here. 

2.5  Benchmarks 

Benehmarks  offer  a  means  to  provide  repeatable  experiments  and  to  provide 
eomparability  between  various  CRs  being  tested.  The  dynamie  interaetion  of  environments, 
goals,  and  eapabilities  of  CRs  means  that  ereating  a  generie  benehmark  is  non-trivial  [24]. 
Eurthermore,  the  performanee  of  a  CR  may  ehange  over  time  as  it  adapts  and  learns. 
However,  the  benefits  of  benehmarking  inelude  providing  a  basis  for  speetrum  regulators 
to  eertify  and  regulate  CR,  vendors  for  type  approval  testing,  and  serviee  providers  for 
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implementation  [24].  The  authors  in  [33]  eoneur,  stating  that  it  is  erueial  for  a  oornmon  set 
of  representative  test  eases  to  be  used  that  every  CR  has  to  pass. 

The  workload  offered  by  a  benehmark  is  the  RF  environment.  Beeause  the  RF 
environment  is  naturally  variant,  multiple  replieations  are  needed  to  aehieve  a  suitable  level 
of  statistieal  eonfidenee  and  to  realize  eomparability  [37]. 

The  eonsensus  among  researehers  is  that  the  best  way  to  test  CR  systems  is  to 
use  representative  eases  of  the  radio  environment  [5,  33,  37].  CREW  [49]  researehers 
propose  having  various  referenee  seenarios  (home,  offiee,  publie  buildings)  and  wireless 
teehnologies  eharaeterized  and  then  use  these  reprodueible  referenee  seenarios  in  tests. 
Both  the  radio  and  network  ean  be  tested  and  metries  eolleeted  this  way.  However, 
erroneous  results  may  be  indieated  when  there  is  observed  external  RF  interferenee.  This 
entire  proeess  ean  eventually  be  automated  to  sweep  the  entire  range  of  interest  [37]. 

2.6  Cognitive  Radio  Test  Methodologies 

A  test  methodology  deseribes  the  overall  proeedure  in  how  a  CR  is  evaluated.  Here, 
strategies  in  implementing  that  overall  proeedure  are  presented.  These  strategies  are  all 
suitable  for  benehmarking. 

2.6.1  Radio  Environment  Map  Scenario  Driven  Testing  Approach. 

Zhao  et  al.  propose  the  use  of  REMs  as  the  means  of  testing  CR  systems  [30].  These 
REMs  represent  seenarios,  and  REM-based  radio  seenario  testing  (REM-SDT)  ean  be 
performed.  This  approaeh  is  transferable  aeross  different  systems,  and  eaeh  system  ean 
ineorporate  varying  levels  of  the  REM  as  neeessary.  Eor  instanee,  the  nominal  REM  is 
a  large  database  eontaining  radio  environment  data  ineluding  geographieal  information, 
serviee  and  networks,  poliey,  deviee  aetivity,  and  so  forth.  A  primitive  CE  may  only  use 
frequeney  information  from  the  REM  in  the  form  of  a  binary  veetor,  while  an  advaneed  CE 
may  use  poliey  and  geographieal  information  in  addition  to  frequeney  information.  REM- 
SDT  enables  both  systems  being  tested  to  use  the  same  input  and  be  evaluated  to  the  same 
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benchmarks.  Furthermore,  these  benchmarks  are  tied  to  both  specific  cases  such  as  IEEE 
802.22  and  to  and  general  case  DSA  environments,  such  that  a  benchmark  suite  is  used  to 
test  CRs  over  a  full  range  of  situations  [24] . 

Also  in  [24],  the  authors  divide  CR  performance  metrics  into  three  levels:  node, 
network,  and  application.  Each  level  provides  different  levels  of  insight  to  the  user,  whether 
they  are  a  regulator,  standards  organization,  or  CR  developer.  The  authors  also  present  the 
idea  of  using  “score-cards”  for  comparability  and  standardization  at  each  level.  The  authors 
next  present  a  means  of  evaluating  CRs  using  game  theory  and  utility  functions. 

The  approach  presented  in  [30]  and  [24]  is  well  suited  for  an  IEEE  802.22-style 
approach,  but  is  not  necessarily  useful  for  other  CR  architectures.  This  is  primarily  because 
the  test  methodology  revolves  around  evaluating  the  CE  and  how  it  processes  REMs. 
Actual  RE  performance  is  not  measured. 

2.6.2  Psychometric  Approach. 

Since  radio  cognition  is  an  analogue  to  human  cognition,  it  is  proposed  to  evaluate 
CR  using  approaches  developed  for  testing  human  cognition  [50].  Psychometric  testing, 
as  described  by  Dietrich  et  ah,  utilizes  item  response  models  (IRMs)  to  assess  CR 
performance  [50].  IRMs  have  been  used  in  psychological  and  educational  fields,  and  depict 
how  latent  traits  govern  behaviors  such  that  the  observed  behaviors  can  be  used  to  estimate 
the  levels  of  those  traits.  Eor  example,  the  latent  trait  of  a  CR  may  be  the  underlying 
CE  algorithm,  while  the  observed  trait  is  the  CR  itself.  CRs  and  CEs  are  assessed  by 
administering  tasks  to  perform.  IRMs  are  constructed  and  applied  to  the  CR  to  elicit  the 
desired  response  from  the  CR.  At  present,  the  psychometric  approach  has  been  limited  to 
evaluation  of  a  subset  of  a  CE  and  not  to  an  entire  system. 

As  with  REM-SDT  testing,  the  psychometric  approach  as  presented  by  [50]  is  geared 
towards  an  IEEE  802.22-style  approach  as  it  focuses  on  evaluating  the  CE. 


27 


2.7  Cognitive  Radio  Prototyping  Platforms 

There  are  a  plethora  of  prototypes  used  in  CR  researeh.  These  prototypes  range  from 
purely  software/simulations  to  hybrid  hardware/software  devices.  Of  interest  are  devices 
that  are  built  using  hardware,  as  this  research  is  focused  on  testing  real  physical  devices. 
Three  main  hardware  prototyping  platforms  are  in  use  by  researchers  today  and  they  are 
presented  here.  All  of  these  platforms  are  similar  in  that  they  enable  researchers  to  rapidly 
prototype  spectrum  sensing,  CE,  and  other  CR  components  as  well  as  offering  control  over 
the  physical  layer.  In  general,  these  platforms  simply  act  as  an  RF  interface  in  which  the 
bulk  of  the  processing  is  done  either  on  an  attached  PC,  though  some  contain  an  embedded 
processor. 

•  USRP:  The  USRP  is  a  SDR  platform  intended  to  be  low-cost  yet  effective  [51].  The 
USRP  is  ideal  for  experiments  with  large  numbers  of  nodes.  The  USRP  offers  a 
flexible  range  of  RF  front-ends  that  enable  operation  from  0  to  6  GHz.  There  are 
several  USRP  variants  available;  in  general,  all  contain  an  on-board  FPGA  for  data 
processing.  USRPs  require  the  use  of  an  external  computing  device,  such  as  a  PC, 
for  full  operation. 

•  WARP:  The  Wireless  Open-Acess  Research  Platform  (WARP)  board  [52,  53]  is  a 
high-performance  SDR  that  enables  user  control  over  all  layers  of  operation.  The 
WARP  is  designed  such  that  the  researcher  has  full  control  over  programming  and 
operation,  thus  suiting  a  wide  range  of  research  needs.  The  WARP  board  has 
up  to  four  RF  interfaces,  though  only  two  are  normally  used.  The  WARP  board 
transmits  using  the  2.4  or  5  GHz  ISM  band.  WARPs  can  fulfill  all  processing  and 
communication  requirements  on  the  board;  however,  it  is  also  possible  to  offload 
processing  and  communication  requirements,  such  as  modulation,  to  an  external 
computing  device. 
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•  Nutaq:  Nutaq  offers  several  diverse  SDRs  for  use  in  CR  applieations  [54].  The 
Nutaqs  take  a  middle  position  between  the  WARP  and  the  USRP:  the  radios  are 
in  general  more  capable  than  USRPs;  however,  they  do  not  offer  the  full  level  of 
control  and  ease  of  use  that  the  WARP  does.  Nutaq  offers  radios  that  can  be  used  in 
embedded  or  development  applications.  The  Nutaq  radios  operate  on  a  wide  range 
of  frequencies  and  not  solely  on  the  2.4  /  5  GHz  ISM  band.  Nutaq  products  are  used 
in  large-scale  CR  testbeds. 

2.8  Cognitive  Radio  Spectrum  Paradigms 

In  DSA,  spectrum  is  used  in  one  of  two  ways  [2].  The  first  is  to  utilize  spectrum 
overlay.  Spectrum  overlay  is  the  basis  for  OSA  and  is  when  SUs  use  spatial  and  temporal 
spectrum  holes.  In  spectrum  overlay,  the  SU  is  free  to  transmit  so  long  as  there  is  no  PU 
transmitting  at  the  same  time.  By  contrast,  spectrum  underlay  allows  the  SU  to  transmit 
concurrent  with  the  PU  so  long  as  the  SU  does  not  interfere  with  PU.  Spectrum  overlay 
and  underlay  are  shown  in  Figure  2.3. 


(a)  Spectrum  Overlay 


(b)  Spectrum  Underlay 


Figure  2.3:  Spectrum  Overlay  and  Underlay  Paradigms 


Both  spectrum  overlay  and  underlay  will  likely  be  used  by  CRSs,  and  thus  both  should 
be  accounted  for  when  developing  a  CR  test  methodology. 
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III.  Proposed  Test  Methodology 


The  proposed  test  methodology,  hereafter  referred  to  as  CRATM,  developed  by  this 
research  is  presented  in  this  chapter.  A  methodology  for  evaluating  CRATM  is 
presented  in  the  next  chapter,  followed  by  the  results  in  the  subsequent  chapter. 

3.1  Overview 

CRATM  is  designed  to  be  an  over- arching  framework  that  applies  to  evaluating  a  CRS. 
CRATM  applies  classes  of  benchmarks  to  stimulate  behaviors  in  the  SUT  (e.g.  a  CRS). 
The  CR  response  to  the  stimuli  is  then  measured  using  performance  metrics.  The  classes 
of  benchmarks  enable  the  testing  to  be  flexible,  yet  allow  comparisons  between  platforms. 
The  radio  environment  is  ideally  controlled  using  emulation.  A  unique  characteristic  of 
CRATM  is  that  the  impact  to  the  PU  by  the  presence  of  a  SU  can  be  measured  directly. 

The  simplest  way  to  view  CRATM  is  that  it  is  similar  in  style  to  computer  benchmarks 
such  as  the  Standard  Performance  Evaluation  Corporation  (SPEC)  suite  [55].  In  SPEC, 
multiple  standard  programs  are  used  to  comprehensively  evaluate  computer  performance. 
Specific  benchmarks  are  comprised  of  programs  that  are  known  to  stress  computer 
performance  while  being  representative  of  expected  workloads.  The  results  from  the 
benchmarks  are  not  necessarily  used  to  gather  data  on  absolute  real-world  performance; 
rather,  the  results  enable  relative  comparison  between  platforms.  However,  because  the 
benchmarks  are  based  on  real-world  problems,  the  results  provide  some  insight  into  real- 
world  behavior. 

Similarly,  the  benchmarks  proposed  in  CRATM  are  not  intended  to  measure  absolute 
real-world  CRS  performance.  Rather,  the  benchmarks  are  proposed  to  allow  an  accurate 
relative  comparison  between  CRSs  as  well  as  to  stimulate  the  CRS  under  test  to  exhibit 
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potentially  undesired  behavior.  This  is  done  by  modeling  the  benehmarks  on  real-world 
radio  environments. 

The  means  of  eontrolling  the  radio  environment  with  CRATM,  and  thus  enabling 
benehmarks,  is  to  use  emulation.  Physieal  CRS  test  deviees  are  used;  however,  the 
signal  paths,  PUs  and  baekground  environment  are  generated  virtually.  The  limitation 
to  this  approaeh  is  that  antennas  must  be  virtualized  as  well;  however,  the  emulation 
environment  offers  flexibility  in  the  implementation  of  antennas.  The  use  of  an  emulated 
RF  environment  is  not  required  for  CRATM,  though  it  is  highly  desired. 

To  test  a  range  of  deviees  from  a  single  CR  node  to  a  network  of  CR  nodes,  sealing  is 
offered  by  the  flexible  use  of  performanee  metries.  The  performanee  metries  eorrespond  to 
partieular  benehmark  elasses.  For  example,  an  evaluation  of  multiple  CR  nodes  operating 
as  a  network  may  use  network  level  performanee  metries,  while  the  evaluation  of  single  CR 
node  may  utilize  node  level  performanee  metries.  Sealing  is  important  to  allowing  broad 
evaluation  of  CRS  platforms  while  still  allowing  eomparison  between  platforms  that  are 
similar. 

The  underlying  assumption  behind  CRATM  is  that  eognition  is  measured  from 
behavior  and  not  by  direet  examination  of  the  system.  This  assumption  is  key  to  enabling 
a  flexible  yet  speeifie  means  of  evaluating  CRSs. 

3.2  Cognitive  Radio  Behavior-Based  Evaluation 

As  stated  previously,  CRATM  assumes  that  a  CRS  ean  be  evaluated  based  on  its 
behavior,  without  knowledge  of  the  speeifie  eognition  proeesses  taking  plaee.  CRATM 
treats  a  deviee  as  eognitive  if  it  performs  the  following  funetions: 

1 .  Improves  performance  by  responding  to  the  environment. 

2.  Avoids  causing  interference  to  existing  users. 

3.  In  performing  1  and  2,  implements  user  policy  and  goals 
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This  definition  can  be  restated  as,  “does  the  device  improve  communications  while 
not  degrading  others’  ability  to  communicate?”.  The  two  key  elements  of  this  definition 
are  that  the  device  improves  throughput  while  not  causing  harm  to  existing  users.  Cognition 
comes  into  play  when  implementing  these  two  objectives.  Therefore,  cognition  does  not 
need  to  be  measured  explicitly  as  it  can  be  measured  through  the  performance  of  improving 
throughput  and  avoiding  causing  interference.  Note  that  this  definition  only  applies  to  DSA 
environments  in  which  spectrum  adaptation  is  essential.  The  cognition  of  a  CRS  cannot  be 
tested  if  it  is  explicitly  instructed  to  transmit  on  open- frequencies  from  an  external  decision 
making  authority.  In  other  words,  a  CRS  should  not  be  tested  using  this  definition  as  if  it 
were  a  licensed  primary  user. 

CRATM  treats  cognition  as  occurring  on  a  continuum  as  opposed  to  occurring  in 
discrete  steps.  Researchers  can  focus  on  evaluating  CRS  performance  without  needing  to 
justify  if  a  device  is  in  fact  cognitive,  or  to  what  relative  cognitive  level  it  is.  With  CRATM, 
cognition  becomes  apparent  when  observing  performance.  A  more  intelligent  cognitive 
CRS  should  perform  better  than  a  less  intelligent  CRS.  The  actual  cognition  levels  of 
the  CRSs  is  secondary  to  performance.  Stated  another  way,  cognitive  ability  only  matters 
if  it  is  manifested  in  performance.  Otherwise,  cognition  is  a  theoretical  abstraction  with 
little  relevance  to  the  real  world.  This  distinction  is  important  as  it  enables  the  evaluation 
of  CRSs  without  being  tied  down  to  specific  definitions  of  cognition.  Furthermore,  this 
distinction  means  that  CRS  devices  can  evolve  from  current  systems  without  requiring  a 
discrete  jump  to  CRS  capability.  Treating  cognition  as  a  continuum  is  a  feature  of  behavior- 
based  testing. 

Behavior-based  testing  is  similar  to  the  psychometric  testing  presented  in  [50]. 
However,  where  psychometric  testing  applies  stimuli  based  on  models  of  the  underlying 
cognitive  behavior,  behavior-based  testing  applies  stimuli  based  on  the  expected  real-world 
environment.  Both  measure  the  observed  behavior  of  the  SUT  and  use  that  information 
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to  extrapolate  the  performanee  of  the  underlying  eognitive  processes.  If  behavior-based 
testing  is  proven  to  work,  it  negates  the  need  to  require  specific  definitions  for  CR. 

3.3  Performance  Metrics 

Based  on  the  definitions  found  in  Section  3.2,  the  behavioral  based  performance 
metrics  available  for  use  fall  into  two  categories:  improving  performance  and  avoiding 
causing  interference.  Within  these  categories,  the  applicable  performance  metric  depends 
on  the  class  of  benchmark  as  presented  in  Section  3.4. 

The  baseline  performance  metrics  for  use  in  measuring  both  improving  performance 
and  avoiding  causing  interference  are  to  measure  BER  and/or  throughput.  For  instance,  the 
SU  BER  may  be  used  as  a  metric  of  improving  performance  while  PU  BER  may  be  used  as 
a  metric  of  avoiding  causing  interference.  Both  SU  and  PU  performance  may  be  baselined 
by  collecting  data  without  the  presence  of  the  other  user.  The  basic  equations  for  BER  and 
throughput  are: 


BER  = 


bit  errors 

total  bits  transmitted 


(3.1) 


,  ,  bits  success  fully  transmitted 

throughput  = - - — ; -  (3.2) 

time  to  transmit  bits 

Other  performance  metrics  that  may  be  used  for  evaluating  both  SU  and  PU 
performance  are  packet  loss/PER,  end-to-end  delay,  delay-variation,  QoS,  or  bandwidth. 

It  is  important  to  note  that  specific  developers  may  have  need  for  a  specific 
performance  metric  besides  the  BER  and  throughput  of  the  PU  and  SU.  Just  as  with 
computer  benchmarks,  the  most  important  benchmark  is  one  that  encompasses  the 
envisioned  end-use  of  the  platform.  Eikewise,  the  most  important  performance  metrics 
(and  benchmarks)  for  a  CRS  derive  from  the  envisioned  end  use  of  the  CRS.  If  power 
conservation  is  paramount,  a  power  metric  can  be  added  and  used.  CRATM  provides  an 
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overarching  framework  for  generic  CRS  testing;  future  testers  can  modify  it  as  they  see  fit 
for  specific  needs. 

3.4  Benchmarks 

Benchmarks  allow  CRS  testing  to  be  flexible,  accurate,  and  repeatable.  CRS  behavior 
is  dependent  on  the  test  scenario.  Similarly,  the  capabilities  of  the  CRS  limit  applicable 
benchmarks.  For  example,  the  benchmark  should  stimulate  the  capabilities  of  a  CRS  to 
sense  the  spectrum.  However,  the  benchmark  should  not  be  used  to  evaluate  a  CRS  if  the 
signal  of  interest  to  detect  lies  outside  of  the  capabilities  of  the  CRS  receiver  test  device. 

To  allow  for  AI  learning  to  occur,  benchmark  scenarios  should  be  programmed  to  be 
time-driven  as  opposed  to  task-driven.  In  other  words,  the  scenarios  are  not  to  be  executed 
as  fast  as  possible  by  the  CRS;  instead,  the  scenarios  operate  on  a  fixed  timeframe  and 
the  performance  metrics  of  the  CRS  are  measured  at  predetermined  time  intervals.  For 
example,  a  CRN  may  be  evaluated  at  one  minute  into  a  scenario,  five  minutes  in,  and  thirty 
minutes  in.  This  provides  time  for  the  CRN  to  adapt  to  the  environment.  A  non-adaptive 
system  may  outperform  the  adaptive  system  initially,  but  the  longer  timespan  provides  an 
opportunity  for  adaptation  to  occur.  In  the  future,  extended  testing  could  feature  scenarios 
that  are  on  the  scope  of  months  to  years.  For  instance,  a  cognitive  engine  may  “learn”  that 
certain  days  of  the  year  have  different  usage  profiles  and  adapt  accordingly. 

Specific  classes  of  benchmarks  enable  a  wide  variety  of  CRS  architectures  to  be 
evaluated.  For  example,  an  IEEE  802.22  CRS  architecture  should  be  evaluated  under 
benchmarks  that  pertain  to  its  architecture.  Various  architectures  are  executed  under  the 
same  benchmark  to  enable  direct  comparison.  If  an  architecture  cannot  be  executed  under 
a  specific  benchmark,  then  it  cannot  be  directly  compared  with  other  architectures  based 
on  that  benchmark.  This  prevents  incorrect  comparative  conclusions  from  being  drawn. 

The  benchmarks  are  split  into  eleven  characteristics,  which  combined,  form  a  unique 
benchmark.  The  top  four  characteristics  can  be  combined  to  form  a  benchmark  class. 
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or  a  grouping  of  similar  benchmarks.  Multiple  benchmark  classes  are  combined  into  a 
benchmark  suite. 

•  Characteristic  1,  Cognitive  Radio  Type:  Cognitive  radio  type  describes  whether  the 
SUT  is  a  specific  CR  device  or  a  CRN.  In  the  case  of  a  CR  device,  the  SUT  is  just 
the  CR  device.  All  other  components  are  either  virtual  or  physical,  but  their  behavior 
is  well  known  and  characterized.  In  the  case  of  a  CRN,  the  SUT  is  comprised  of  a 
network  of  CR  devices.  All  devices  on  the  network  are  considered  part  of  the  SUT, 
unless  otherwise  explicitly  stated  for  the  particular  benchmark. 

•  Characteristic  la,  Number  of  Nodes:  If  the  SUT  is  a  CRN,  the  number  of  nodes  must 
be  specified.  To  simplify  benchmark  classification,  the  number  of  nodes  may  be  set 
to  common  numbers  of  interest  such  as  2,  4,  10,  or  100  nodes. 

•  Characteristic  2,  Topology:  The  topology  drives  the  type  of  scenario  implemented. 
The  topologies  are  distributed,  centralized,  and  combined.  For  example,  an  IEEE 
802.22  CR  should  not  be  evaluated  in  a  distributed  environment  as  it  not  is  designed 
for  that  environment.  Distributed  environments  imply  no  centralized  base  station, 
while  centralized  environments  do.  The  advantage  in  offering  this  distinction  is  that 
it  provides  flexibility  in  the  type  of  architectures  tested.  A  CRN  may  be  tested  with  a 
virtual  base  station  instead  of  requiring  a  physical  base  station.  Making  the  topology 
a  key  characteristic  enables  specificity  when  comparing  architectures. 

•  Characteristic  3,  Radio  Frequency  Environment:  The  RE  environment  describes  the 
emulated  RE  environment.  The  options  include,  but  are  not  limited  to,  urban,  rural, 
suburban,  home,  and  battlefield  environments.  The  RE  environment  may  also  be  a 
combination  of  these  environments. 
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•  Characteristic  4,  Test  Band:  The  test  band  lays  out  the  overall  frequeney  range 
eligible  to  be  reaehed  by  the  SUT.  The  emulated  RF  environment  is  generated  sueh 
that  PUs  appear  in  this  band  as  specified  by  the  scenario  description. 

•  Characteristic  5,  Motion:  The  SUT  node(s)  may  be  either  static  or  moving.  If 
moving,  the  node(s)  follow  a  preprogrammed  route  that  is  defined  for  the  benchmark. 
Otherwise,  the  node(s)  remain  fixed  at  predetermined  locations. 

•  Characteristic  6,  Performance  Metrics:  The  performance  metrics  of  interest  are 
defined  under  this  characteristic.  The  metrics  are  drawn  from  those  defined  in  Section 

3.3. 

•  Characteristic  7,  Timeframe  and  Sampling  Intervals:  The  overall  test  duration  and 
sampling  intervals  are  defined  in  this  characteristic. 

•  Characteristic  8,  Geographical  Information:  The  locations  of  each  node  plus 
virtualized  units  are  described. 

•  Characteristic  9,  Maximum  Power  Level:  The  maximum  power  level  provides  an 
upper  bound  on  the  CR  power  levels  so  that  performance  metrics  are  not  artificially 
boosted.  It  is  specific  to  each  scenario.  In  practice,  the  maximum  power  levels  may 
be  drawn  from  FCC  regulations. 

•  Characteristic  10,  Primary  User  Profile(s):  The  characteristics  of  the  PUs  are 
specified  here.  Parameters  include  frequency,  power,  location,  duty  cycle,  and 
waveform,  among  others.  The  PU  profiles  form  the  core  workload  of  the  benchmark. 

•  Characteristic  II,  Scenario  Description:  The  final  characteristic  is  a  detailed 
scenario  description.  Any  information  not  already  detailed  out  above  must  be 
categorized  here.  For  example,  if  a  priori  spectrum  knowledge  is  assumed,  this 
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knowledge  should  be  speeified  here.  Then,  that  information  is  passed  to  the  SUT 
when  loading  the  benehmark. 

An  example  benehmark  based  on  the  benehmark  eharaeteristies  is  shown  in  Table  3.1. 


Table  3.1:  Example  CRATM  Benehmark 


Charaeteristie 

Deseription 

1 

CR  Type 

CRN 

la 

Number  of  Nodes 

4 

2 

Topology 

Distributed 

3 

RF  Environment 

Rural 

4 

Test  Band 

2.4  -  2.5  GHz 

5 

Motion 

Statie 

6 

Performanee  Metries 

SU  -  BER,  PU  -  Throughput 

7a 

Timeframe 

15  minutes 

7b 

Sampling  Interval 

1,5,  15  minutes 

8 

Geographieal  Information 

4x4  node  eentered  at  90,  90,  0 

9 

Maximum  Power  Eevel 

-40  dBm 

10 

Primary  User  Profile 

802.1  In  Wireless  Router 

eonneeted  to  4  laptop  deviees 
with  nominal  paeket  traffie 


1 1  Seenario  Deseription  Ad-hoe  test.  No  apriori 

signal  knowledge.  BPSK  PU 
at  2.462  GHz  at  -50  dBm. 
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3.5  Means  of  Testing 

The  means  of  testing  for  CRATM  is  to  utilize  an  emulated  RF  environment  in  which 
all  activity  from  CR  RF  output  to  CR  RF  input  is  controlled  via  the  emulation  environment. 
This  necessitates  using  virtualized  antennas.  Virtual  PUs  are  created  within  the  emulated 
environment,  negating  the  need  for  physical  PU  devices  (though  they  are  allowed,  if 
necessary).  This  enables  rapid  changeover  between  benchmarks.  All  elements,  except 
for  the  physical  SUT  devices,  are  software  defined.  CRATM  may  be  implemented  without 
using  an  emulated  RF  environment;  however,  flexibility  in  running  different  benchmarks  is 
lost. 


3.6  Test  Framework 

The  proposed  CRATM  test  framework  is  as  follows.  An  emulated  radio  environment 
is  created  which  consists  of  PUs  with  various  waveforms  that  communicate  dynamically 
over  the  physical  layer.  These  virtual  PUs  are  designed  to  mimic  either  expected  real- 
world  conditions  or  to  show  specific  test  conditions.  This  emulated  radio  environment 
of  virtual  PUs  acts  as  a  repeatable  benchmark  so  that  different  CRS  radios  are  compared 
by  running  the  same  benchmark.  Physical  CRs,  shielded  from  each  other,  are  connected 
to  the  test  system  and  evaluated  by  collecting  performance  metrics.  Full  spectrum 
knowledge  is  known  and  is  controlled  since  the  emulation  happens  at  the  physical 
layer.  Furthermore,  since  the  physical  layer  emulation  is  accurate,  realistic  higher-layer 
operation  of  both  virtual  PUs  and  the  physical  test  devices  is  expected.  Finally,  since 
the  emulated  environment  is  repeatable,  test  results  are  comparative  between  different 
CRSs.  The  physical  test  devices  are  treated  as  “black  boxes”  in  that  the  behavior  of  the 
device  is  measured  via  device  outputs  as  opposed  to  monitoring  and  measuring  the  actual 
internal  processes  of  the  device.  The  emulated  environment  allows  a  rapid  turnaround 
of  the  test  environment  as  compared  to  field  testing,  while  offering  more  realism  than 
laboratory/simulation  testing. 
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3.7  Comparison  to  the  Literature 

CRATM  is  tested  to  see  if  it  contains  characteristics  and  attributes  of  proposed  test 
methodologies  as  found  in  the  literature.  If  CRATM  does  not  contain  those  attributes,  then 
justification  is  required  to  show  that  CRATM  is  sufficient.  In  other  words,  if  there  are  any 
novel  components  to  CRATM  that  are  not  found  in  published  data,  they  should  be  justified 
through  further  research. 

Data  for  this  literature  comparison  are  the  characteristics  and  attributes  of  proposed 
test  methodologies  as  collected  from  published  peer-reviewed  journals  and  articles. 
Secondary  data  sources  are  conference  workshop  proceedings  and  theses.  Since  the  field 
of  CR  is  relatively  new,  data  is  not  excluded  unless  it  is  specifically  refuted  in  subsequent 
published  work  or  is  of  marginal  relevance.  Conversely,  data  is  given  greater  weight  if  it  is 
cited  or  repeated  by  subsequent  work. 

3.7.1  Key  Characteristics. 

The  collected  data  is  categorized  by  the  following  characteristics: 

•  Means  of  Testing:  Since  the  scope  of  this  research  is  for  testing  physical  CR  devices, 
the  means  of  testing  includes  laboratory,  emulation,  and  real-world.  Laboratory 
testing  may  include  testing  inside  of  a  controlled  environment  such  as  an  anechoic 
chamber.  Emulation  testing  includes  testing  in  which  the  RF  path  is  emulated. 
Real-world  testing  includes  testing  of  devices  in  uncontrolled  and  controlled  RF 
environments.  The  difference  between  real-world  and  laboratory  testing  is  one  of 
degree.  Faboratory  testing  allows  fine-grain  control  of  the  RF  environment  and  near¬ 
ideal  operation  of  equipment  while  real-world  testing  offers  coarse-grain  control  of 
the  RF  environment  (if  at  all)  and  non-ideal  equipment  operation. 

•  Performance  Metrics:  Performance  metrics  are  used  to  evaluate  CR  performance  at 
both  the  node  and  network  levels. 
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•  Attributes:  The  attributes  of  the  data  are  elements  such  as  reproducibility  and 
accuracy. 

•  System  Under  Test:  The  SUT  is  used  to  define  the  test  device.  These  attributes  are 
derived  from  the  various  CR  architectures. 

•  Cognition  Evaluation  Methodology:  The  overall  approach  and  strategy  to  evaluating 
cognitive  ability. 

3.7.2  Results. 

The  literature  was  surveyed  and  the  results  are  summarized  in  Table  3.2.  All 
references  in  the  bibliography,  except  hardware  and  technical  specification  references,  were 
considered  when  compiling  the  categories  and  attributes  in  the  table. 

The  following  elements  were  found  in  the  aggregate  data  that  are  not  found  in 
CRATM:  open-air  testing,  directly  measuring  spectrum  sensing,  and  directly  measuring 
cognitive  ability.  Also,  neither  cognitive  methodology,  psychometric  or  REM-SDT,  in  the 
literature  is  utilized  in  CRATM.  Open-air  testing  can  be  viewed  as  an  either/or  requirement; 
that  is,  if  emulation  meets  the  requirement,  then  open-air  testing  is  not  required.  Thus,  this 
does  not  need  to  be  justified.  However,  the  remaining  discrepancies  do  need  to  be  justified. 

CRATM  does  not  directly  measure  spectrum  sensing  or  cognitive  ability.  As  argued  in 
Section  3.2,  behavior-based  testing  is  proposed  to  take  the  place  of  measuring  spectrum 
sensing  and  cognitive  ability  directly.  Behavior-based  testing  also  takes  the  place  of 
psychometric  testing  and  REM-SDT  testing.  A  methodology  to  evaluate  the  hypothesis 
that  behavior-based  testing  measures  cognitive  ability  is  presented  in  the  next  chapter. 

3.8  Summary 

CRATM  offers  a  means  to  evaluate  the  performance  of  a  CRS,  acting  as  a  SU,  in  a 
controlled  yet  realistic  environment.  Workloads  offered  to  the  test  device  are  classified  into 
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Table  3.2:  Comparison  Between  CRATM  and  Published  CR  Test  Methodology  Attributes 


Category 

Attribute 

Does  CRATM  Contain? 

Means  of  Testing 

Emulation 

Y 

Open-Air 

N 

Performance  Metrics 

Avoid  Causing  Interference 

Y 

Spectrum  Sensing 

N 

Cognitive  Ability 

N 

Network/Device  Performance 

Y 

Attributes 

Reproducability 

Y 

Accuracy 

Y 

Uses  Benchmarks 

Y 

Test  Overlay 

Y 

Test  Underlay 

Y 

SUT  Characteristics 

Single  Device 

Y 

Heterogeneous  Network 

Y 

Homogeneous  Network 

Y 

Centralized  Architecture 

Y 

Distributed  Architecture 

Y 

Cognitive  Methodology 

Psychometric 

N 

REM-SDT 

N 

benchmarks  in  a  manner  similar  to  SPEC  benchmarks  used  with  modern  computers.  CRSs 
are  evaluated  using  benchmarks  so  that  the  results  are  comparative  across  multiple  types 
of  systems.  Ideally,  the  PU  is  created  in  an  emulated  RF  environment  to  allow  greater 
flexibility  in  testing,  though  it  is  not  necessary.  The  key  feature  that  CRATM  offers  is  the 
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capability  to  measure  SU  cognition  by  measuring  its  behavior  and  performance.  CRATM 
assumes  that  eognition  is  defined  as  the  ability  to  improve  SU  performanee  while  not 
negatively  impacting  the  PU,  all  while  implementing  user  goals  and  poliey.  However,  this 
is  yet  to  be  proven.  Furthermore,  CRATM  is  unique  in  that  it  proposes  to  measure  the 
impaet  to  the  PU  direetly  by  measuring  PU  performanee. 
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IV.  Methodology 


4.1  Problem  Background 

The  key  systemic  problem  facing  CR  developers  and  researchers  is  how  to  validate  CR 
performance  such  that  its  real-world  behavior  is  testable  and  knowable  outside  of 
the  real-world.  CR  performance,  by  definition,  is  subject  to  change  and  evolve  over  time. 
A  derivative  problem  facing  CR  researches  is  how  to  test  real  hardware  CRSs  in  order  to 
evaluate  end-to-end  CR  performance.  The  means  of  testing  the  CR  must  be  repeatable, 
accurate,  and  meaningful  so  that  results  are  extendable  to  real-world  applications.  CRATM 
does  not  address  the  former  problem;  however,  it  does  address  the  latter. 

4.1.1  Goals  and  Hypotheses. 

The  goal  of  this  research  is  to  develop  a  test  framework  such  that  given  a  hardware 
CRS  to  test,  the  CRS  can  be  successfully  evaluated  so  that  the  evaluation  results  are 
comparative  to  other  CRSs  (CRSs  may  be  comprised  of  either  a  single  CR  node  or  a 
network  of  CR  nodes  as  defined  in  2.1).  CRATM  is  proposed  to  fulfill  this  research  goal. 
As  there  is  currently  no  standardized  method  of  testing  CR  systems,  CRATM  serves  as  a 
proof  of  concept.  CRATM  cannot  be  evaluated  in  its  entirety  by  measuring  its  utility  against 
established  test  methodologies,  nor  can  it  be  evaluated  by  exhaustively  gathering  data  on  all 
elements  of  CRATM  due  to  a  shortage  of  CR  prototypes.  Instead,  the  novel  component  of 
CRATM,  that  SU  cognition  may  be  determined  by  measuring  behavior,  is  evaluated.  This 
behavior-based  testing  component  is  critical  to  the  success  or  failure  of  CRATM.  Thus,  if  it 
shown  to  be  false,  CRATM  is  not  sufficient  in  its  current  form.  However,  if  it  is  not  shown 
to  be  false,  then  further  development  and  testing  of  CRATM  is  warranted.  The  formal 
hypothesis  to  be  evaluated  is  as  follows: 


•  Hypothesis:  The  cognition  of  cognitive  radio  systems  can  be  successfully  tested  by 
measuring  cognitive  radio  behavior  of  the  entire  system,  without  knowledge  of  the 
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behavior  of  the  underlying  cognitive  components.  The  null  hypothesis  is  that  the 
eognition  of  eognitive  radio  systems  ean  only  be  evaluated  by  direet  measurement 
and  analysis  of  the  eognitive  eomponents. 

The  hypothesis  tests  the  key  underlying  assumption  of  CRATM.  This  underlying 
assumption  is  that  a  CR  is  defined  to  be  eognitive  if  it  performs  three  speeifie  tasks  in 
a  DSA  environment:  1)  improves  performanee,  2)  avoids  eausing  interferenee  to  existing 
users,  and  3)  implements  user  poliey  and  goals  when  performing  1  and  2.  If  this  assumption 
is  true,  then  it  is  possible  to  measure  CRS  behavior  without  needing  to  direetly  evaluate  the 
eognition  of  speeifie  eomponents.  The  hypothesis  is  tested  by  eomparing  CR  prototypes 
with  non-CR  prototypes,  with  the  expeetation  that  CRs  with  the  full  OODA  eyele  (sense  the 
speetrum,  reaet  to  it,  and  ehange  waveforms  as  required)  will  perform  better  than  deviees 
without  the  full  OODA  eyele. 

4.1.2  Approach. 

The  first  step  in  this  researeh  is  to  develop  a  CR  prototype  to  serve  as  a  CRS  SUT. 
The  CR  prototype  performs  all  four  aetions  of  the  OODA  eyele  as  deseribed  in  Seetion  2.2. 
After  basie  CR  deviee  performanee  has  been  validated,  the  hypothesis  is  tested  as  follows. 

A  PU  network  is  implemented  alongside  a  SU  network.  The  PU  is  used  to  ereate  the 
environment  sensed  by  the  SU.  The  SU  has  a  CE  with  speetrum  sensing  eapability.  The 
PU  and  SU  are  setup  to  operate  simultaneously,  and  performanee  metries  are  eolleeted  for 
both.  Various  desired  waveforms  (user  goals)  are  passed  to  the  SU  to  serve  as  a  baseline 
when  the  CE  is  off.  These  experiments  are  repeated  with  the  SU  CE  on.  The  results  should 
show  that  the  PU  and  SU  performanee  improves  when  the  SU  CE  is  on  as  eompared  to 
when  it  is  off.  Eurthermore,  the  PU  and  SU  performanee  should  improve,  or  stay  eonstant, 
as  greater  levels  of  eognitive  adaptability  are  added  to  the  SU.  Cognitive  adaptability  for 
this  experiment  is  inereased  waveform  ehoiees  and  bandwidth  for  the  SU.  It  is  assumed 


44 


for  this  experiment  that  greater  adaptability  in  waveform  and  bandwidth  is  correlated  with 
greater  cognitive  ability. 

The  PU  implements  several  different  waveforms  in  order  to  provide  diversity  in 
spectrum  opportunities.  For  example,  one  experiment  workload  may  be  a  PU  that  uses 
a  constant  wide -band  transmission,  while  the  next  experiment  workload  may  be  a  PU  that 
uses  a  frequency  hopping  narrow-band  transmission.  A  range  of  PU  waveforms  are  used 
so  that  the  SU  response  to  the  PU  created  environment  may  be  adequate  characterized. 

All  experimental  configurations  are  repeated  for  the  case  when  the  opposing  network 
is  off.  In  other  words,  the  experiments  are  repeated  with  the  SU  off  and  PU  on  and  then 
with  the  PU  off  and  the  SU  on.  These  experiments  serve  as  a  control. 

4.2  System  Boundaries 

The  SUT,  shown  in  Figure  4.1,  includes  a  wireless  radio  node  pair  as  part  of  a  small 
communications  network.  One  node  is  a  designated  transmitter  while  the  other  node 
is  the  receiver.  For  this  experiment,  the  SUT  is  limited  to  WARP  nodes  [52]  running 
WARPLab  [56].  In  WARPLab,  the  radio  node  only  serves  as  a  transmitter  or  receiver 
and  all  modulation  or  demodulation  of  the  waveform  takes  place  on  board  a  PC  connected 
via  Ethernet. 

The  component  under  test  (CUT)  is  the  CE,  or  the  ability  of  the  radio  to  execute  the 
OODA  cycle.  Eor  the  baseline  non-cognitive  system,  the  CUT  is  the  unmodified  WARP 
design,  which  has  no  observe,  orient,  or  decision  part  of  the  OODA  cognition  cycle.  Eor 
the  cognitive  system,  the  WARP  design  is  modified  to  use  an  energy-detection  spectrum 
sensing  algorithm  to  implement  the  full  OODA  cycle.  Both  the  cognitive  and  non-cognitive 
configurations  use  the  same  hardware  configuration.  The  software  is  identical  with  the 
exception  of  an  additional  function,  the  CE,  added  to  the  CR  to  sense  the  environment  and 
cognitively  choose  a  waveform. 
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System  Parameters 


Figure  4.1:  DSA  Testbench  SUT 


The  SUT  is  limited  to  transmission  on  a  single  2.4  GHz  ISM  band  channel.  Within 
this  20  MHz  channel,  the  spectrum  is  broken  down  into  128  sub-channels,  or  frequency 
bins.  The  SUT  uses  only  64  of  those  bins,  for  a  bandwidth  of  10  MHz. 

For  the  purposes  of  this  experiment,  the  SUT  is  also  referred  to  as  the  SU.  The  scope  of 
the  SUT  is  limited  to  the  cognitive  engine  presented  in  Section  4.6  using  WARP  hardware. 

4.3  System  Services 

The  SUT  offers  two  services:  communication  and  non-interference.  These  services 
are  derived  from  the  hypothesis.  Communication  applies  to  the  SU,  while  non-intererence 
applies  to  the  impact  of  the  SU  on  the  PU. 

Communication  (or  throughput)  is  defined  as  the  successful  transmission  of  data  from 
one  node  and  its  successful  receipt  by  other  nodes.  Due  to  the  scope  of  the  SUT,  only 
physical  layer  data  communication  is  tested  and  the  data  of  interest  being  transmitted  is 
only  bits.  The  outcomes  of  this  service  are  communication  success,  partial  success  or 
failure.  Success  occurs  when  all  data  is  successfully  transmitted  between  the  transmitter 
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and  receiver.  Partial  success  occurs  when  data  is  transmitted  between  the  transmitter  and 


receiver  but  is  degraded  due  to  interference  from  the  RF  environment.  Failure  occurs  when 
data  is  not  transmitted  due  to  interference. 

The  non-interference  service  is  the  ability  of  the  SU  nodes  to  not  interfere  with  a 
PU.  Interference  occurs  when  there  is  simultaneous  transmission  of  a  SU  and  a  PU  on 
the  same  frequency  bin.  There  are  two  types  of  interference:  closed  channel  interference 
and  open  channel  interference.  In  closed  channel  interference,  a  channel  is  occupied  by 
a  PU  prior  to  the  transmission  of  interference  by  a  SU.  Successful  avoidance  of  closed 
channel  interference  means  that  the  SU  does  not  transmit  during  any  time  when  the  PU  is 
transmitting  for  a  channel.  Failure  is  when  the  SU  does  transmit  on-top  of  an  existing  PU. 
In  open  channel  interference,  a  channel  is  open  and  used  by  a  SU,  but  is  then  occupied  by  a 
PU.  The  SU  will  inadvertently  cause  interference  in  this  case,  so  the  goal  is  to  minimize  the 
time  it  takes  for  the  SU  to  detect  that  it  is  interfering  with  a  PU  and  abandon  the  channel. 

4.4  Workload 

The  workload  for  the  SUT  is  PU  activity.  The  PU  activity  creates  the  radio 
environment  that  enables  the  SUT  to  react  and  adapt  to  in  accordance  with  its  CE.  The 
workload  is  generated  by  two  WARP  radio  nodes  running  WARPLab.  One  PU  node  is  the 
transmitter  and  the  other  PU  node  is  the  receiver.  There  is  only  one  PU  pair  (transmitter 
and  receiver)  for  this  experiment.  The  PU  transmitter  is  used  to  create  a  total  of  five 
environments. 

The  SU  may  use  spectrum  holes  created  by  the  PU,  as  shown  in  Figure  4.2.  In  this 
figure,  the  10  MHz  spectrum  bandwidth  is  broken  up  into  64  frequency  bins  that  the  SU 
may  use  to  transmit  on.  Spectrum  holes  are  indicated  as  light  areas  on  the  REM  plot.  An 
arbitrary  spectrum  sensor  threshold  line  is  also  shown  on  the  chart  to  signify  how  the  SU 
determines  what  frequencies  are  occupied  or  not. 
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Power  Spectral  Density  (dB/Hz) 


Figure  4.2:  PU  Created  Speetrum  Holes 
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4.5  Performance  Metrics 


The  performance  metrics  are  derived  from  the  system  services  and  are  chosen 
in  accordance  with  the  metrics  found  in  Section  3.3.  For  this  experiment,  both  SU 
communication  performance  and  PU  non-interference  are  measured  using  Equation  4. 1 . 


Throughput  — 


Number  of  Attempts  to  Receive  *  Bits  Correctly  Demodulated  *  WARP  Buffer  S  ize 

WARP  Clock  Frequency 


(4.1) 


The  units  of  Equation  4.1  are  in  bits  per  second  (bps).  The  throughput  of  the  SU  and 
the  throughput  of  the  PU  are  measured  independently. 

A  secondary  performance  metric  used  in  this  experiment  is  BER,  as  shown  in  Equation 
4.2.  All  modulation  schemes  used  in  this  research  use  binary  modulation.  Therefore,  BER 
is  used  as  opposed  to  the  more  generic  symbol  error  rate  (SER). 


Number  o  f  Bits  in  Error 

BER  = - ^ ^ - -  (4.2) 

T otal  Number  of  Bits  T ransmitted 

BER  is  used  to  corroborate  throughput  data.  Eor  greater  SU  cognition  ability,  there 
should  be  an  increase  in  throughput  for  both  the  PU  and  SU.  If  the  throughput  cannot 
increase  due  to  a  bandwidth  limitation,  then  the  BER  should  be  seen  to  decrease  or 
stay  constant.  BER  is  not  used  as  the  primary  metric  as  it  does  not  fully  characterize 
performance.  Eor  example,  if  the  CE  user  goal  is  to  minimize  BER,  then  a  minimal 
bandwidth  waveform  will  be  chosen.  However,  the  throughput  will  be  drastically  reduced, 
limiting  its  desirability  for  use  in  real-world  communications. 

4.6  System  Parameters 

The  system  parameters  are  characteristics  that  affect  the  performance  of  the  SUT. 
Initial  pilot  experiments  reduced  the  number  of  key  system  parameters  down  to  the 
following  list. 
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•  Desired  Waveform:  The  desired  radio  waveform  type  impaets  the  eommunieations 
capabilities  of  the  SUT  as  well  as  the  ability  of  the  SUT  to  avoid  interfering  with  PUs. 
Three  specific  waveforms  are  used  in  this  experiment:  frequency  shift  keying  (FSK), 
orthogonal  frequency  division  multiplexing  (OFDM),  and  dis-contiguous  OFDM 
(d-OFDM).  All  waveforms  are  strictly  physical  layer  only  and  no  MAC  is  used. 
The  bandwidth  of  the  waveform  is  directly  correlated  to  a  number  of  frequency  bins. 
For  example,  an  OFDM  waveform  using  16  bins  (contiguous)  will  have  an  overall 
bandwidth  of  16  *  ^  =  2.5  MHz,  where  Fs  is  the  sampling  frequency  of  40  MHz.  In 
the  case  that  the  CE  is  off,  the  desired  waveform  is  in  fact  the  waveform  that  is  used  in 
the  experiment.  In  the  case  that  the  cognitive  engine  is  on,  the  cognitive  engine  will 
try  to  maximize  its  performance  by  selecting  the  desired  waveform;  however,  due 
to  environmental  constraints,  a  lower  performing  waveform  may  be  utilized.  FSK 
uses  binary  modulation,  and  the  OFDM  waveforms  use  binary  phase-shift  keying 
modulation. 

•  Transmission  Power:  The  radio  node  transmission  power  inversely  affects  communi¬ 
cations  performance  and  non-interference.  In  general,  increasing  transmission  power 
to  improve  communications  performance  will  negatively  impact  the  non-interference 
capability.  For  this  research,  the  transmission  power  is  kept  constant  and  is  not  var¬ 
ied  by  the  cognitive  engine.  The  transmit  gain  on  the  MAX2829  transceiver  on  the 
WARP  is  set  to  0.  The  receive  gain  on  the  MAX2829  transceiver  is  set  to  15. 

•  Cognitive  Engine  Algorithm:  The  CE  allows  a  radio  node  to  optimize  communica¬ 
tions  performance  and  non-interference.  The  cognitive  engine  for  this  experiment 
is  a  simple  bandwidth  optimizing  algorithm.  Based  on  the  sensed  environment,  the 
algorithm  chooses  the  best  waveform  that  maximizes  SU  throughput  while  avoid¬ 
ing  transmitting  on  frequency  bins  occupied  by  the  PU.  Eor  instance,  the  user  may 
desire  a  OEDM  waveform  with  16  bins  of  bandwidth.  The  sensed  spectrum,  how- 
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ever,  reveals  that  there  are  only  8  eontiguous  frequeney  bins  available.  As  8  bins  is 
the  transition  between  FSK  and  OFDM,  the  algorithm  preseribes  downgrading  the 
waveform  to  FSK  (as  it  has  superior  throughput  for  bandwidth  of  8  bins  or  less).  The 
algorithm  is  presented  in  Figure  4.3. 

It  is  important  to  note  that  this  CE  algorithm  is  intended  to  represent  the  baseline 
funetionality  of  a  CE  and  not  neeessarily  to  be  a  fully  funetional,  state  of  the  art  CE. 
In  Mitola’s  eognition  levels  [6],  this  CE  algorithm  would  likely  lie  around  Cognitive 
Level  1  or  2.  For  referenee.  Cognitive  Level  1  is  Goal-Driven,  meaning  that  the  CR 
ehooses  a  waveform  aeeording  to  a  goal.  Cognitive  Level  2  is  Context  Awareness, 
meaning  that  the  CR  has  knowledge  of  what  the  user  is  trying  to  do. 

Based  on  pilot  experiments,  the  system  is  most  sensitive  to  the  CE  status  (on  or  off)  as 
well  as  to  the  ehoiee  of  desired  radio  waveform. 

4.7  Workload  Parameters 

The  workload  parameters  are  workload  eharaeteristies  that  affeet  the  performanee  of 
the  SUT. 

•  Power  and  Position:  The  transmission  power,  as  well  as  relative  position,  of  eaeh 
PU  impaets  the  pereeived  speetrum  of  eaeh  individual  SUT  node.  To  simplify 
the  experiment,  the  PU  transmission  power  remains  eonstant  and  equal  for  all 
PUs.  Furthermore,  the  position  of  the  PUs  to  the  SUs  is  fixed  on  a  2  x  2 
reetangular  grid  with  dimensions  1 8  inehes  by  36  inehes.  This  orientation  was  ehosen 
primarily  for  a  eonfiguration  that  ean  be  replieated  in  the  DYSE,  but  also  due  to 
equipment  eonstraints  and  to  minimize  the  proximity  of  the  antennas  to  the  radios. 
It  should  be  noted  that  pilot  experiments  did  not  show  an  appreeiable  difference  for 
various  antenna  configurations.  The  PU  transmission  power  is  set  equal  to  the  SU 
transmission  power  of  0  dB  gain  for  the  MAX2829  chip  on  the  WARP  radios. 
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Figure  4.3:  Cognitive  Engine  Algorithm  Flowehart 
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•  Waveform:  The  PU  waveform  is  either  a  wideband,  narrowband,  or  dis-eontiguous 
(non-eontiguous)  signal  to  ereate  a  range  of  RF  environments.  These  speeifie 
waveforms  are  ereated  using  FSK,  OFDM,  or  d-OFDM.  The  overall  waveform  type 
stays  eonstant  within  an  experiment,  though  the  frequeneies  it  transmits  on  may 
ehange. 

•  Transmission  Duration:  For  each  experiment,  the  PU  is  set  to  transmit  for  a  minimum 
duration.  The  minimum  transmission  duration  acts  as  a  discrete  step  size.  For 
instance,  the  total  experiment  time  may  be  60  seconds  and  the  minimum  transmission 
duration  is  1  second.  If  the  PU  waveform  utilizes  frequency  hopping,  then  the 
frequency  will  change  once  every  second. 

The  total  time  to  transmit  one  WARPLab  buffer  over  the  air  is  approximately  410 
/rs.  Data  processing  was  found  to  take  two  to  three  orders  of  magnitude  longer. 
Therefore,  to  minimize  the  total  experimental  time  and  to  drastically  reduce  the 
number  of  experiments  required,  the  PU  transmitter  is  set  to  continuously  transmit. 
The  transmit  buffer  is  updated  for  each  new  BER  calculation  cycle  and  the  transmit 
waveform  is  updated  at  each  transmission  duration  interval. 

•  External  RF  Environment:  The  RF  environment  is  either  emulated  or  over-the-air. 
For  RF  environment  emulation,  testing  takes  place  in  the  DYSE.  All  antennas  are 
assumed  to  be  dipole  antennas.  Eor  over-the-air,  testing  takes  place  on  a  lab  bench. 
Antennas  are  located  on  a  plane  and  are  separated  by  at  least  3  feet  from  the  nearest 
WARP  board.  Antennas  are  dipole  antennas. 

To  simplify  the  experiment,  the  workload  parameters  are  combined  into  a  single 
workload  factor,  which  is  referred  to  as  the  Environment.  This  is  discussed  further  in 
the  following  section. 
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4.8  Factors 


Factors  are  the  parameters  that  are  varied  during  the  experiment  and  are  derived  from 
the  parameters.  The  factors  for  this  experiment  are  summarized  in  Table  4.1. 


Table  4.1: 

SUT  Eaetors 

Type 

Eactor 

Eevels 

System 

Desired  Waveform 

ESK-1  Bin  BW,  ESK-8  Bins  BW 

OEDM- 16  Bins  BW,  OEDM-48  Bins  BW 

System 

Cognitive  Engine 

Off,  On 

Workload 

Environment 

1,2,  3,  4,  5 

Workload 

External  RE  Environment 

Eab,  Emulated 

4.8.1  System  Factors. 

•  Desired  Waveform:  Four  different  waveforms  levels  are  available  for  this  experiment. 
These  are  a  FSK  waveform  with  1  bin  of  bandwidth,  a  FSK  waveform  with  8  bins  of 
bandwidth  (BW),  a  eontiguous  OFDM  waveform  with  16  bins  of  bandwidth,  and  a 
contiguous  OFDM  waveform  with  48  bins  of  bandwidth.  For  readability,  waveforms 
are  referred  to  as  the  modulation-bandwidth  as  in  FSK-8.  These  four  levels  serve  as 
maximum  performanee  boundaries  for  when  the  CE  is  on.  For  instanee,  the  CE  when 
on  may  ehoose  a  lower  performing  waveform  for  use  than  the  desired  waveform.  The 
performanee  without  interferenee,  however,  will  not  be  any  greater  than  the  desired 
waveform  with  the  CE  off.  OEDM-48  is  the  highest  performing  waveform.  Dis¬ 
contiguous  OEDM  has  lower  performanee  than  contiguous  OEDM. 

•  Cognitive  Engine:  The  SU  operates  with  the  CE  on  or  off.  If  off,  the  SU  randomly 
generates  a  new  waveform  (in  aeeordanee  with  the  desired/speeified  waveform)  in 
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place  of  each  CE  cycle.  For  example,  consider  a  case  where  the  desired  waveform 
is  FSK  with  1  bin  bandwidth.  With  the  CE  on,  the  CE  may  be  programmed  to  take 
0.25  seconds  per  sense-process-eommunicate  (i.e.  OODA)  eycle.  With  the  CE  off,  a 
new  frequeney  bin  to  transmit  on  is  seleeted  every  0.25  seeonds. 

4.8.2  Workload  Factors. 

The  workload  parameters,  except  external  RE  environment,  are  combined  to  form  a 
single  workload  faetor,  referred  to  as  the  environment. 

•  Environment:  Five  distinet  Environments  are  ereated  for  this  experiment:  a  constant 
FSK  tone  with  8  bins  BW,  a  hopping  FSK  tone  with  8  bins  BW,  a  eontiguous 
OFDM  transmission  with  32  bins  BW  that  has  a  eonstant  bandwidth,  a  non¬ 
contiguous  OFDM  transmission  with  48  bins  BW  that  changes  waveform  every  5 
transmission  eyeles,  and  a  randomized  non-contiguous  OFDM  waveform  with  16 
bins  of  bandwidth.  These  environments  are  depicted  graphieally  in  Figure  4.4. 

Note  that  the  frequeney  bins  utilized  for  transmission  are  randomly  generated  for 
eaeh  experiment.  The  transmission  duration  is  fixed  at  1  seeond  and  the  total 
experiment  time  is  fixed  at  60  seconds. 

•  External  RE  Environment:  The  environment  is  either  the  lab  environment  or  the 
emulated  environment  of  the  DYSE. 

4.9  Evaluation  Technique 

The  experiment  is  run  on  four  WARP  v2  hardware  platforms.  Two  radios  are 
designated  for  use  as  the  PU  pair,  while  the  others  are  designated  for  use  as  the  SU 
pair.  The  antennas  and  radios  are  plaeed  such  that  each  pair  communicates  aeross  the 
diagonal  as  shown  in  Figure  4.5.  Each  radio  pair  is  controlled  by  one  HP  Faptop  running 
MATFAB  R2012b  and  WARPFab  v7.3.  Both  laptops  are  identical  images  operating  on  64- 
bit  Windows  7.  The  laptops  are  HP  Compaq  8510p  with  Intel  Centrino  proeessors.  Both 
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20  40  60 

Frequency  Bin 


20  40  60 

Frequency  Bin 


20  40  60 

Frequency  Bin 


(a)  Environment  1 


(b)  Environment  2 


(c)  Environment  3 


Frequency  Bin  Frequency  Bin 


(d)  Environment  4  (e)  Environment  5 

Figure  4.4:  PU  Created  Workload  Environments.  Clockwise  from  top-left:  FSK-1,  FSK- 
8,  OFDM-32  Contiguous,  OFDM-48  Non-contiguous,  and  OFDM- 16  Randomized.  Note 
that  dark  areas  indicates  frequencies  that  the  PU  transmits  on. 
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laptops  and  the  radios  are  eonneeted  via  Ethernet  to  a  Ciseo  Gb  switeh.  The  RF  ehannel  is 
approximately  white  Gaussian  noise. 


PU  Node  1 


SU  Node  2 


PU  Node  2 


SU  Node  1 


MATLAB 


Gigabit  Switeh 


PU  Controller 


SU  Controller 


Figure  4.5:  Experimental  Setup 


The  PU  laptop  aets  as  a  eontroller  for  the  experiment.  A  list  of  experimental 
eonfigurations  are  loaded  onto  the  PU  laptop.  Then,  the  order  of  experiments  is  randomly 
sorted.  The  PU  laptop  shares  the  eurrent  experiment  eonfiguration  with  the  SU  laptop. 
Prior  to  eaeh  experiment,  the  speetrum  sensing  thresholds  for  the  SU  are  ealibrated  in  order 
to  minimize  environmental  effeets.  Calibration  is  performed  by  having  the  PU  transmit  a 
known  waveform  and  then  lowering  the  threshold  level  on  the  SU  speetrum  sensor  until  the 
measured  REM  oeeupies  at  least  all  of  the  bins  for  the  PU  transmitted  waveform.  Then, 
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the  PU  transmits  a  second  known  waveform  and  the  threshold  level  is  checked  for  validity. 
If  there  are  a  greater  number  of  REM  bins  occupied  by  interference  than  by  the  desired  PU 
signal,  the  calibration  process  is  repeated. 

Following  calibration,  the  laptops  synchronize  and  then  the  PU  laptop  sends  a  trigger 
signal  to  the  SU  laptop,  upon  which  both  laptops  begin  executing  the  experiment  for  a 
specified  duration.  Once  the  time  for  each  experiment  has  expired,  output  parameters  and 
performance  metrics  are  saved  and  the  process  begins  again. 

Each  experiment  is  programmed  to  be  60  seconds  in  length.  Pilot  experiments 
evaluated  experimental  durations  of  10,  30,  60,  and  120  seconds.  These  experiments 
showed  that  60  seconds  was  sufficient  to  minimize  variance  of  the  results. 

The  results  of  this  experiment  serve  as  a  proof  of  concept.  However,  the  results  are 
also  validated  since  BER  is  calculated  alongside  throughput.  Since  the  transmission  powers 
and  ambient  noise  are  known  via  use  of  a  spectrum  analyzer,  the  performance  of  the  SU  and 
PU  pairs  may  be  validated  against  known  data.  If  the  computed  BER  is  outside  expected 
bounds,  then  the  experimental  results  may  not  be  valid. 

The  experimental  setup  as  used  with  the  DYSE  is  identical  except  that  rather  than 
communicating  over  the  air,  the  WARP  boards  are  connected  via  SMA  cable  to  the  DYSE 
RE  inputs.  Additionally,  the  SU  transmit  node  is  replicated  in  software  in  the  DYSE. 

4.10  Experimental  Design 

A  full  factorial  experimental  design  is  used,  excluding  the  RE  environment  factor. 
That  is,  there  are  4  x  2  x  5  =  40  experimental  trials.  Additionally,  baseline  performance 
data  is  collected  for  each  factor,  which  adds  another  8  SU  factor  levels  +  5  PU  factor  levels 
=  13  trials.  Therefore,  there  are  a  total  of  53  trials  conducted  for  one  experiment  set.  A 
minimum  of  20  repetitions  are  accomplished  in  order  to  sufficiently  reduce  95%  confidence 
intervals. 
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The  results  are  assessed  by  utilizing  both  a  two-sided  t-test  as  well  as  by  using 
a  scoring  system.  In  both  cases,  the  results  should  show  that  for  all  PU  workload 
environments,  the  SU  configuration  with  the  cognitive  engine  on  improves  both  PU  and 
SU  throughput  when  compared  to  when  the  cognitive  engine  is  off.  Furthermore,  the 
results  should  show  that  PU  and  SU  throughput  is  improved  as  greater  levels  of  cognition 
are  added  (i.e.  using  higher  performing  waveforms).  If  these  results  are  found,  then  the 
hypothesis  will  have  been  shown  to  not  be  false.  This  research  is  not  intended  to  prove  that 
the  hypothesis  is  true;  rather,  the  research  is  intended  to  see  if  the  hypothesis  is  false. 

4.11  Methodology  Summary 

The  hypothesis  is  designed  to  test  whether  or  not  a  cognitive  radio  can  be  evaluated 
by  measuring  its  behavior.  The  SUT,  which  is  the  SU,  is  a  WARP  radio  pair  that  has 
cognitive  capability.  The  SU  is  evaluated  against  a  PU  workload,  which  drives  the  SU.  The 
throughput  of  both  the  SU  and  PU  are  gathered  to  assess  communications  throughput  of 
the  SU  and  non-interference  to  the  PU. 
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V.  Results  and  Analysis 


The  results  of  the  experiments  deseribed  in  the  previous  ehapter  are  diseussed  here.  An 
analysis  demonstrating  that  the  eolleeted  data  is  valid  is  presented  in  Appendix  A. 

5.1  Evaluating  Behavior  Based  Cognitive  Radio  Testing 

The  hypothesis  is  designed  to  test  whether  or  not  a  eognitive  radio  ean  be  evaluated 
by  measuring  its  behavior.  The  results  of  evaluating  this  hypothesis  are  presented  in  this 
seetion.  The  expeeted  results  are  that  the  SU  throughput  is  inereased  when  the  CE  is  on 
relative  to  when  the  CE  is  off,  that  the  PU  throughput  is  inereased  when  the  SU  CE  is  on 
relative  to  when  the  SU  CE  is  off,  and  that  greater  CE  funetionality  will  result  in  greater 
throughput  for  both  the  SU  and  PU.  CE  funetionality  is  assumed  to  be  based  on  the  ability 
of  the  SU  to  adapt.  Since  all  SU  configurations  with  the  CE  on  use  the  same  underlying 
algorithm,  the  adaptation  component  comes  from  the  choice  of  the  waveform.  As  presented 
in  Section  4.8.1,  OEDM  waveforms  are  considered  more  adaptive  and  capable  than  ESK 
waveforms,  meaning  that  the  OEDM  waveforms  are  considered  more  cognitively  useful 
for  the  purposes  of  this  experiment.  The  order  of  cognition  goes  from  ESK-1,  ESK-8, 
OEDM- 16,  OEDM-48. 

The  experiments  were  collected  in  the  ACRO  lab.  All  experimental  data  was  collected 
using  over-the-air  RE  transmissions.  A  total  of  25  to  30  data  points  were  collected  for 
each  experimental  configuration.  An  attempt  was  made  to  run  the  experiments  in  the 
dynamic  spectrum  environment  emulator  (DYSE);  however,  this  attempt  did  not  produce 
useful  results. 

The  throughput  of  testing  each  PU  environment  is  shown  in  Eigures  5.1,  5.2,  and 
5.3.  The  charts  are  interpreted  as  follows.  The  top  subplot  shows  the  SU  throughput  for 
each  CE  configuration,  while  the  bottom  chart  shows  the  PU  throughput.  Error  bars  are 
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shown  for  each  bar  in  the  plot.  The  magnitude  of  the  SU  throughput  is  primarily  affected 
by  the  desired  waveform  and  CE  status  (the  x-axis).  Smaller  deviations  in  SU  throughput 
are  due  to  it  being  affected  by  the  specific  PU  created  environment.  The  PU  throughput 
magnitude  is  primarily  affected  by  the  choice  of  the  PU  environment,  though  the  choice 
of  the  SU  waveform  and  CE  status  provides  a  smaller  effect.  Eor  the  SU  throughput,  the 
experimental  limit  line  for  each  CE  configuration  is  the  maximum  performance  measured 
for  that  CE  and  waveform  configuration  with  the  PU  turn  off.  Eor  the  PU  throughput,  the 
experimental  limit  line  is  the  maximum  measured  PU  performance  with  the  SU  turned  off. 

The  remainder  of  this  chapter  examines  the  effect  of  the  SU  CE  on  SU  throughput 
and  BER,  the  effect  of  the  SU  CE  on  PU  throughput  and  BER,  the  overall  rank  of  each  CE 
using  a  scoring  system,  and  an  examination  on  the  validity  of  the  data.  Einally,  this  chapter 
presents  the  data  collection  attempt  using  the  DYSE. 

5.2  Effect  of  Cognitive  Engine  on  Secondary  User  Throughput  and  BER 

A  t-test  comparison  of  the  SU  throughput  values  is  shown  in  Table  5.1.  The  t-test 
utilized  is  a  single  sided  t-test  to  evaluate  whether  or  not  the  SU  throughput  with  the  CE 
on  is  higher  than  when  the  CE  is  off.  Eor  example,  the  t-test  evaluates  whether  or  not  the 
throughput  of  OEDM-16  with  the  CE  on  is  greater  than  the  throughput  of  OEDM-16  with 
the  CE  off.  The  t-test  significance  level  used  is  Qr=0.05. 

The  CE  improves  SU  throughput  in  55%  of  cases  when  compared  to  the  corresponding 
waveform  with  the  CE  off.  The  exceptions  are  the  cases  where  the  PU  utilizes  a  relatively 
large  amount  of  the  bandwidth,  as  with  Environments  3,  4  and  5.  Eor  these  exceptions, 
the  throughput  with  the  CE  off  is  greater  for  one  primary  reason.  This  is  because  the 
CE  on  condition  means  that  the  waveform  is  adapting  to  the  environment;  as  a  result, 
the  bandwidth  available  to  the  SU  decreases  and  so  the  throughput  decreases  as  well. 
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Table  5.1:  Secondary  User  Throughput 


Environment 

Waveform 

Mean  Throughput  (bps) 

CE  Off  CE  On 

t-test:  CE  Off 

less  than  CE  On? 

p-value 

1 

FSK  1 

72566.5 

73181.12 

Yes 

2.86E-13 

1 

FSK8 

586072.1 

588475.3 

Yes 

6.97E-06 

1 

OFDM  16 

1959149 

2126356 

Yes 

3.29E-07 

1 

OFDM  48 

5917864 

6859417 

Yes 

6.48E-17 

2 

FSK  1 

72477.17 

73178.99 

Yes 

6.07E-13 

2 

FSK  8 

586129.3 

588055.3 

Yes 

3.58E-07 

2 

OFDM  16 

1957118 

2101196 

Yes 

6.89E-36 

2 

OFDM  48 

5882772 

6911066 

Yes 

3.85E-59 

3 

FSK  1 

72651.49 

73178.46 

Yes 

3.37E-11 

3 

FSK  8 

588245.1 

585617.6 

No 

1 

3 

OFDM  16 

2076489 

2282354 

Yes 

0.000986 

3 

OFDM  48 

5765735 

3716908 

No 

1 

4 

FSK  1 

72416.31 

71440 

No 

0.989317 

4 

FSK  8 

588485.9 

131121.8 

No 

1 

4 

OFDM  16 

2088617 

137588.4 

No 

1 

4 

OFDM  48 

6122345 

156274.4 

No 

1 

5 

FSK  1 

72515.63 

73152.62 

Yes 

3.51E-10 

5 

FSK  8 

588330.6 

534137 

No 

1 

5 

OFDM  16 

2072105 

1971300 

No 

0.999768 

5 

OFDM  48 

5947066 

2436589 

No 

1 
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By  comparison,  the  throughput  of  the  SU  with  the  CE  on  is  greater  for  all  narrowband 
interferenee  eonditions  (Environments  1  and  2). 

Next,  the  performanee  of  the  CE  is  evaluated  with  respeet  to  relative  CE  eapability. 
These  results  only  apply  to  the  eondition  when  the  SU  CE  is  on.  ESK-1  is  assumed  to  be 
the  least  eognitive  CE  eonfiguration,  while  OEDM-48  is  assumed  to  be  the  most  eognitive 
CE  eonfiguration.  Eor  this  analysis,  eaeh  CE  eonfiguration  is  eompared  to  all  previous 
eonfigurations  to  evaluate  whether  it  is  statistieally  greater  or  statistieally  less  (using  t- 
test,  cr=0.05).  If  the  differenee  is  neither  greater  nor  less,  then  the  eonfigurations  are 
eonsidered  to  have  equal  performanee.  The  results  of  this  analysis  are  presented  in  Table 
5.2.  Waveform  1  is  the  waveform  with  greater  adaptation,  waveform  2  is  the  waveform 
with  relatively  lower  adaptation. 

The  throughput  inereases  for  greater  eognition  levels  in  all  Environments  exeept  for 
one  ease.  This  one  exeeption  shows  that  in  PU  Environment  4,  OEDM  16  has  performanee 
that  is  not  statistieally  different  than  the  ESK  8  performanee.  In  no  eases  is  the  throughput 
less  for  greater  eognition  levels. 

The  BER,  whieh  is  equivalent  to  the  SER  for  the  waveforms  in  this  experiment, 
is  shown  in  Table  5.3  along  with  assoeiated  statisties.  In  this  table,  the  BER  statisties 
are  eomputed  between  experiments,  thus  treating  eaeh  experimental  result  as  a  random 
variable.  Note  that  these  statisties  only  apply  to  the  experimental  eondition  where  both  the 
PU  and  SU  are  on.  The  BER  of  the  SU  is  shown  graphieally  in  Eigure  5.4. 

Relative  to  the  SU  waveform  with  the  CE  off,  the  BER  deereases  at  a  statistieally 
signifieant  level  (Qr=0.05)  when  the  CE  is  on  75%  of  the  time.  Of  the  remaining  25%, 
two  oases  have  p-values  between  0.05  and  0.95,  indioating  that  they  are  not  statistieally 
signifieant.  This  leaves  three  of  the  oomparisons,  or  15%,  to  not  follow  the  trend  of 
deereasing  the  BER  when  the  CE  is  turned  on.  These  eases  are  for  Environment  4/ESK-l, 
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Table  5.2:  Relative  Comparison  of  Seeondary  User  Throughput  with  CE  On 


Environment 

Waveform  1 

Waveform  2 

Waveform  1  Throughput 

Greater  than  Waveform  2? 

p-value 

Waveform  1  Throughput 

Less  than  Waveform  2? 

p-value 

1 

FSK8 

FSK  1 

Yes 

1.2E-142 

No 

1 

1 

OFDM  16 

FSK  1 

Yes 

3.55E-56 

No 

1 

1 

OFDM  16 

FSK  8 

Yes 

l.OlE-49 

No 

1 

1 

OFDM  48 

FSK  1 

Yes 

1.02E-65 

No 

1 

1 

OFDM  48 

FSK  8 

Yes 

8.27E-64 

No 

1 

1 

OFDM  48 

OFDM  16 

Yes 

7.82E-51 

No 

1 

2 

FSK8 

FSK  1 

Yes 

8.9E-142 

No 

1 

2 

OFDM  16 

FSK  1 

Yes 

8.3E-112 

No 

1 

2 

OFDM  16 

FSK  8 

Yes 

1.79E-96 

No 

1 

2 

OFDM  48 

FSK  1 

Yes 

2.7E-111 

No 

1 

2 

OFDM  48 

FSK  8 

Yes 

6.8E-101 

No 

1 

2 

OFDM  48 

OFDM  16 

Yes 

1.4E-99 

No 

1 

3 

FSK8 

FSK  1 

Yes 

l.lE-127 

No 

1 

3 

OFDM  16 

FSK  1 

Yes 

1.93E-35 

No 

1 

3 

OFDM  16 

FSK  8 

Yes 

3.2E-30 

No 

1 

3 

OFDM  48 

FSK  1 

Yes 

3.86E-67 

No 

1 

3 

OFDM  48 

FSK  8 

Yes 

9.95E-64 

No 

1 

3 

OFDM  48 

OFDM  16 

Yes 

4.1E-25 

No 

1 

4 

FSK8 

FSK  1 

Yes 

3.28E-18 

No 

1 

4 

OFDM  16 

FSK  1 

Yes 

1.09E-18 

No 

1 

4 

OFDM  16 

FSK  8 

No 

0.165192 

No 

0.834808 

4 

OFDM  48 

FSK  1 

Yes 

1.43E-16 

No 

1 

4 

OFDM  48 

FSK  8 

Yes 

0.002125 

No 

0.997875 

4 

OFDM  48 

OFDM  16 

Yes 

0.015524 

No 

0.984476 

5 

FSK8 

FSK  1 

Yes 

1.66E-72 

No 

1 

5 

OFDM  16 

FSK  1 

Yes 

2.61E-55 

No 

1 

5 

OFDM  16 

FSK  8 

Yes 

7.01E-50 

No 

1 

5 

OFDM  48 

FSK  1 

Yes 

1.59E-54 

No 

1 

5 

OFDM  48 

FSK  8 

Yes 

3.19E-50 

No 

1 

5 

OFDM  48 

OFDM  16 

Yes 

1.13E-14 

No 

1 
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Table  5.3:  Secondary  User  BER 


Environment 

Waveform 

Mean  BER 

t-test:  CE  On 

p-value 

CE  Off 

CE  On 

less  than  CE  Off? 

1 

ESK  1 

0.006614 

0 

Yes 

6.4381  lE-13 

1 

ESK8 

0.004244 

0.000106 

Yes 

1.74591E-08 

1 

OEDM  16 

0.071575 

0.007902 

Yes 

2.35172E-15 

1 

OEDM  48 

0.101613 

0.04094 

Yes 

2.78773E-10 

2 

ESK  1 

0.007182 

0 

Yes 

7.41619E-13 

2 

ESK  8 

0.0039 

0.000507 

Yes 

3.4321E-14 

2 

OEDM  16 

0.074598 

0.017047 

Yes 

1.1527E-41 

2 

OEDM  48 

0.106953 

0.049613 

Yes 

1.10588E-47 

3 

ESK  1 

0.005432 

0 

Yes 

8.0045E-11 

3 

ESK  8 

0.000003 

0 

No 

0.160970674 

3 

OEDM  16 

0.014988 

0.001915 

Yes 

1.15547E-17 

3 

OEDM  48 

0.124882 

0.017339 

Yes 

6.08817E-37 

4 

ESK  1 

0.008076 

0.018283 

No 

0.989102325 

4 

ESK  8 

0 

0.004765 

No 

0.999699766 

4 

OEDM  16 

0.010543 

0.005217 

Yes 

0.00028507 

4 

OEDM  48 

0.070659 

0.013046 

Yes 

1.69936E-29 

5 

ESK  1 

0.006912 

0.000327 

Yes 

6.18991E-10 

5 

ESK  8 

0 

0.000023 

No 

0.911885749 

5 

OEDM  16 

0.019015 

0.037046 

No 

1 

5 

OEDM  48 

0.097331 

0.052722 

Yes 

9.06373E-41 
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Figure  5.4:  Secondary  User  BER 
(Lines  and  Bars  not  visible  are  near  or  equal  to  0) 
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Environment  4/FSK-8,  and  Environment  5/OEDM-16.  The  latter  ease  is  not  eonsequential 
as  the  OEDM  waveforms  do  not  neeessarily  perform  reliably.  That  is,  pilot  experiments 
found  that  non-eontiguous  waveforms  have  a  signifieantly  higher  BER  than  eontiguous 
waveforms.  Sinee  all  oases  using  OEDM  with  the  CE  off  are  eontiguous,  this  means  that  it 
is  both  possible  and  likely  for  the  CE  off  to  have  a  lower  BER  as  the  CE  on  eondition 
will  likely  be  non-eontiguous  due  to  its  adapting  to  the  non-eontiguous  environment. 
This  leaves  two  anomalous  data  points:  Environment  4/ESK-l  and  Environment  4/ESK- 
8.  However,  the  effeet  of  the  CE  on  PU  throughput  must  also  be  eonsidered. 

5.3  Effect  of  Cognitive  Engine  on  Primary  User  Throughput  and  BER 

A  t-test  eomparison  of  the  PU  throughput  values  is  shown  in  Table  5.4.  As  with  the  SU 
throughput  eomparison,  the  t-test  utilized  is  a  single  sided  t-test  to  evaluate  whether  or  not 
the  PU  throughput  with  the  CE  on  is  higher  than  when  the  CE  is  off.  The  t-test  signifieanee 
level  used  is  Qr=0.05. 

The  CE  improves  PU  throughput  in  every  ease  exeept  for  one,  whieh  is  Environment 
2/OEDM-48.  In  this  ease,  there  is  statistieally  no  differenee  between  the  performanee  of 
the  two  CE  eonditions.  The  PU  throughput  never  deereases  with  the  CE  on  as  eompared  to 
the  CE  off. 

As  with  the  SU  throughput,  the  performanee  of  the  CE  is  evaluated  with  respeet  to 
relative  CE  eapability.  Again,  these  results  only  apply  to  the  eondition  when  the  SU  CE 
is  on.  Eor  this  analysis,  eaeh  CE  eonfiguration  is  eompared  to  all  previous  eonfigurations 
to  evaluate  whether  it  is  statistieally  greater  or  statistieally  less  (using  t-test,  Q'=0.05).  If 
the  differenee  is  neither  greater  nor  less,  then  the  eonfigurations  are  eonsidered  to  have 
equal  performanee.  The  results  of  this  analysis  are  presented  in  Table  5.5.  Waveform  1  is 
the  waveform  with  greater  adaptation,  waveform  2  is  the  waveform  with  relatively  lower 
adaptation. 
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Table  5.4:  Primary  User  Throughput 


Environment 

Waveform 

Mean  Throughput  (bps) 

CE  Off  CE  On 

t-test:  CE  Off 

less  than  CE  On? 

p-value 

1 

FSK  1 

552154.1 

589581.2 

Yes 

9.04E-34 

1 

FSK8 

547076.2 

589512.3 

Yes 

2.59E-34 

1 

OFDM  16 

582573.1 

590011.1 

Yes 

1.42E-13 

1 

OFDM  48 

589703.8 

589871.7 

Yes 

0.024796 

2 

FSK  1 

555733.9 

582969.4 

Yes 

1.67E-32 

2 

FSK  8 

545570.3 

581846.5 

Yes 

4.41E-34 

2 

OFDM  16 

581621.5 

588271.4 

Yes 

2.07E-24 

2 

OFDM  48 

589780.6 

589836.6 

No 

0.252609 

3 

FSK  1 

3494620 

4176330 

Yes 

1.63E-30 

3 

FSK  8 

3230874 

4052200 

Yes 

7.39E-35 

3 

OFDM  16 

3446377 

4148722 

Yes 

2.05E-35 

3 

OFDM  48 

2561634 

4145628 

Yes 

2.64E-43 

4 

FSK  1 

4427361 

4748057 

Yes 

9.43E-15 

4 

FSK  8 

4092883 

4855452 

Yes 

2.96E-32 

4 

OFDM  16 

4632167 

4887000 

Yes 

4.94E-09 

4 

OFDM  48 

3581395 

4867963 

Yes 

9.69E-47 

5 

FSK  1 

1110428 

1192591 

Yes 

3.99E-22 

5 

FSK  8 

1062312 

1153178 

Yes 

7.29E-26 

5 

OFDM  16 

1096236 

1215137 

Yes 

1.63E-26 

5 

OFDM  48 

900189.2 

1209777 

Yes 

1.04E-63 
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Table  5.5:  Relative  Comparison  of  Primary  User  Throughput  with  CE  On 


Environment  Waveform  1  Waveform  2  Waveform  1  Throughput  p-value  Waveform  1  Throughput  p-value 

Greater  than  Waveform  2?  Less  than  Waveform  2? 


1 

FSK8 

FSK  1 

No 

0.609159 

No 

0.390841 

1 

OFDM  16 

FSK  1 

Yes 

0.015775 

No 

0.984225 

1 

OFDM  16 

FSK  8 

Yes 

0.012376 

No 

0.987624 

1 

OFDM  48 

FSK  1 

No 

0.052527 

No 

0.947473 

1 

OFDM  48 

FSK  8 

Yes 

0.036262 

No 

0.963738 

1 

OFDM  48 

OFDM  16 

No 

0.900236 

No 

0.099764 

2 

FSK8 

FSK  1 

No 

0.934301 

No 

0.065699 

2 

OFDM  16 

FSK  1 

Yes 

1.78E-13 

No 

1 

2 

OFDM  16 

FSK  8 

Yes 

6.35E-19 

No 

1 

2 

OFDM  48 

FSK  1 

Yes 

7.95E-18 

No 

1 

2 

OFDM  48 

FSK  8 

Yes 

2.78E-23 

No 

1 

2 

OFDM  48 

OFDM  16 

Yes 

3.03E-15 

No 

1 

3 

FSK8 

FSK  1 

No 

0.999832 

Yes 

0.000168 

3 

OFDM  16 

FSK  1 

No 

0.878119 

No 

0.121881 

3 

OFDM  16 

FSK  8 

Yes 

0.000317 

No 

0.999683 

3 

OFDM  48 

FSK  1 

No 

0.912021 

No 

0.087979 

3 

OFDM  48 

FSK  8 

Yes 

0.000238 

No 

0.999762 

3 

OFDM  48 

OFDM  16 

No 

0.659105 

No 

0.340895 

4 

FSK8 

FSK  1 

Yes 

0.004328 

No 

0.995672 

4 

OFDM  16 

FSK  1 

Yes 

0.001667 

No 

0.998333 

4 

OFDM  16 

FSK  8 

No 

0.244949 

No 

0.755051 

4 

OFDM  48 

FSK  1 

Yes 

0.000683 

No 

0.999317 

4 

OFDM  48 

FSK  8 

No 

0.363533 

No 

0.636467 

4 

OFDM  48 

OFDM  16 

No 

0.672912 

No 

0.327088 

5 

FSK8 

FSK  I 

No 

1 

Yes 

1.41E-10 

5 

OFDM  16 

FSK  1 

Yes 

6.57E-05 

No 

0.999934 

5 

OFDM  16 

FSK  8 

Yes 

1.98E-17 

No 

1 

5 

OFDM  48 

FSK  1 

Yes 

0.000187 

No 

0.999813 

5 

OFDM  48 

FSK  8 

Yes 

5.42E-20 

No 

1 

5 

OFDM  48 

OFDM  16 

No 

0.881703 

No 

0.118297 
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The  PU  throughput  does  not  always  inerease  for  greater  cognition  levels.  However, 
the  vast  majority  of  these  cases  are  not  statistically  significant;  that  is,  the  PU  throughput 
does  not  decrease  for  greater  cognition  levels.  However,  there  are  two  cases  in  which 
the  PU  throughput  does  decrease  for  greater  cognition  levels:  Environment  3/FSK-8  & 
FSK-1  and  Environment  5/ESK-8  &  ESK-1.  In  both  of  these  cases,  the  ESK-8  waveform 
negatively  impacts  the  PU  compared  to  the  FSK-1  waveform.  Interestingly,  the  FSK-8 
does  not  improve  PU  throughput  for  any  environment  except  for  Environment  4.  Possible 
factors  could  be  a  failure  of  the  CE  to  properly  sense  the  environment,  the  selection  of  a 
sub-optimal  channel  by  the  CE,  or  additional  interference  caused  by  the  FSK-8  waveform 
apart  from  channel  selection.  The  first  two  factors  were  investigated  and  not  found  to  be 
factors  in  and  of  themselves.  Analysis  of  the  problem  revealed  that  the  side-lobe  power 
was  not  considered  in  development  of  the  CE  due  to  power  not  being  considered  a  relevant 
factor.  In  reality,  the  FSK  waveforms  do  not  transmit  solely  on  the  desired  frequencies 
and  have  non-negligible  sidelobes.  The  additional  power  coming  from  the  sidelobes  is 
more  likely  to  cause  interference  to  the  PU  with  wider  bandwidth  SU  waveforms.  This  is 
consistent  with  the  results  found. 

Also,  it  is  important  to  note  that  the  actual  waveform  used  by  the  SU  may  be  different 
than  the  desired  waveform.  For  example,  if  the  largest  spectrum  hole  created  by  the  PU 
is  less  than  8  bins  wide,  the  CE  FSK  waveform  will  downgrade  from  8  to  4,  2,  or  1  bins 
of  bandwidth.  This  may  account  for  the  Environment  4  being  different  than  the  other 
environments  with  respect  to  the  FSK  8  /  FSK  1  relationship. 

The  BER  and  associated  statistics  are  shown  in  Table  5.6.  The  BER  of  the  PU  is  shown 
graphically  in  Figure  5.5. 

In  every  case,  the  PU  has  a  lower  BER  when  the  SU  CE  is  on  as  compared  to  when  it 
is  off.  The  two  anomalous  data  points  in  which  the  SU  BER  was  worse  with  the  CE  on  still 
exhibit  the  desired  effect  towards  the  PU. 
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Table  5.6:  Primary  User  BER 


Environment 

Waveform 

Mean  BER 

CE  Off  CE  On 

t-test:  CE  On 

less  than  CE  Off? 

p-value 

1 

ESK  1 

0.063888 

0.000509 

Yes 

7.81E-34 

1 

ESK8 

0.072066 

0.000668 

Yes 

1.02E-34 

1 

OEDM  16 

0.012551 

0.00002 

Yes 

1.32E-13 

1 

OEDM  48 

0.000283 

0.000002 

Yes 

4.04E-11 

2 

ESK  1 

0.057951 

0.011575 

Yes 

2.47E-32 

2 

ESK  8 

0.074967 

0.013682 

Yes 

3.42E-34 

2 

OEDM  16 

0.014407 

0.002826 

Yes 

6.57E-29 

2 

OEDM  48 

0.000273 

0.00006 

Yes 

1.75E-12 

3 

ESK  1 

0.180992 

0.023933 

Yes 

3.78E-46 

3 

ESK  8 

0.24082 

0.049386 

Yes 

1.03E-42 

3 

OEDM  16 

0.191427 

0.023141 

Yes 

9.29E-42 

3 

OEDM  48 

0.404031 

0.02501 

Yes 

5.89E-51 

4 

ESK  1 

0.238317 

0.18244 

Yes 

1.08E-18 

4 

ESK  8 

0.29457 

0.166341 

Yes 

1.81E-35 

4 

OEDM  16 

0.206524 

0.160484 

Yes 

1.9E-12 

4 

OEDM  48 

0.385684 

0.162651 

Yes 

4.67E-50 

5 

ESK  1 

0.178626 

0.117131 

Yes 

1.93E-37 

5 

ESK  8 

0.21625 

0.148447 

Yes 

9.51E-37 

5 

OEDM  16 

0.189113 

0.102721 

Yes 

3.59E-45 

5 

OEDM  48 

0.337774 

0.111745 

Yes 

1.28E-68 
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Figure  5.5:  Primary  User  BER 
(Lines  and  Bars  not  visible  are  near  or  equal  to  0) 
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5.4  Scoring 

CRATM  proposes  to  use  benehmarks  and  relative  seoring  to  evaluate  CRSs.  A 
preliminary  means  of  seoring  was  applied  against  the  experimental  results.  The  seoring 
presented  here  refleets  the  data  analysis  already  presented  and  is  not  intended  for  drawing 
eonelusions  on  the  hypothesis.  Rather,  the  seoring  system  is  presented  to  show  how 
CRATM  eaptures  the  experimental  results.  Seores  are  based  on  data  means. 

For  eaeh  environment,  the  highest  performing  SU  eonfiguration  (e.g.  SU  CE  on, 
OFDM  48  waveform)  is  assigned  a  seore  of  10,  as  measured  by  throughput.  The  lowest 
performing  SU  eonfiguration  is  assigned  a  0.  All  other  data  points  are  assigned  a  value 
between  0  and  10  based  on  their  relative  value  to  the  minimum  and  maximum.  This  ranking 
is  applied  to  both  the  SU  throughput  and  the  PU  throughput.  Then,  the  SU  throughput  seore 
and  the  PU  throughput  seore  are  averaged  together  by  a  speeified  weight  to  produee  the  final 
seore. 

The  final  seore  refleets  the  performance  of  both  the  SU  and  PU  combined  into  a  single 
metric.  Using  the  CRATM  CR  definition,  the  most  cognitive  configuration  should  have  the 
highest  score  while  the  least  cognitive  configuration  should  have  the  lowest  score.  This 
is  not  a  rigorous  evaluation  of  cognition  due  to  the  limitations  of  the  SUT;  however,  it 
provides  a  rough  means  to  correlate  the  experimental  results  with  CRATM. 


Table  5.7:  Performance  Scoring,  SU  Weight=50%  PU  Weight=50% 


Environment  FSK 1 

CE  Off 

ESK8 

CE  Off 

OEDM  16 

CE  Off 

OEDM  48 

CE  Off 

ESK  1 

CE  On 

ESK  8 

CEOn 

OEDM  16 

CE  On 

OEDM  48 

CE  On 

1 

0.2 

0 

5.4 

9.3 

4.8 

5.1 

6.4 

10 

2 

0.8 

0 

5.3 

9.2 

4 

4.3 

6.2 

10 

3 

0.7 

0 

3.5 

4.4 

4.4 

4.6 

7.8 

10 

4 

2.6 

0 

10 

7.9 

6.3 

7.7 

8.1 

7.9 

5 

0.8 

0 

4.6 

5.1 

4.2 

3.6 

9.2 

10 
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Table  5.7  shows  the  seoring  using  a  50/50%  weighting.  Table  5.8  shows  the  results 
from  changing  the  weighting  to  25/75%,  where  75%  of  the  weight  is  accorded  to  PU 
throughput. 


Table  5.8:  Performance  Scoring,  SU  Weight=25%  PU  Weight=75% 


Environment  FSK 1 

CE  Off 

ESK8 

CE  Off 

OEDM  16 

CE  Off 

OFDM  48 

CE  Off 

FSK  1 

CE  On 

FSK  8 

CEOn 

OFDM  16 

CE  On 
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In  the  50/50  case,  there  is  an  upward  trend  in  scores  for  greater  levels  of  cognition 
when  the  CE  is  on.  The  one  exception  to  this  trend  is  from  a  case  where  the  SU 
performs  better  with  increased  cognition,  but  the  PU  performance  difference  is  statistically 
insignificant.  Furthermore,  the  CE  on  condition  scores  higher  than  the  CE  off  condition 
except  for  Environment  4.  As  discussed  previously,  in  this  environment  the  SU  with  the 
CE  on  effectively  constrains  itself  to  not  cause  interference  to  the  PU,  meaning  that  the  SU 
throughput  drops  by  a  large  amount  relative  to  the  CE  off  condition.  In  the  25/75  case,  the 
same  trends  are  evident  except  that  the  CE  on  condition  always  performs  better  than  the 
corresponding  CE  off  condition. 

For  future  scoring,  actual  weights  would  be  set  to  desired  benchmark  conditions.  For 
example,  if  PU  non-interference  is  a  key  performance  parameter,  then  the  weight  set  for 
that  metric  would  be  set  accordingly  higher. 
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5.5  DYSE 


Data  collection  was  attempted  using  the  DYSE.  A  virtual  PU  transmitter  was  created 
in  the  DYSE  that  transmitted  a  recurring  bit-sequence  to  a  physical  PU  receiver.  A  physical 
SU  transmitter  and  receiver  communicated  through  the  DYSE  RE  interface.  Unfortunately, 
the  RE  signals  in  and  out  of  the  DYSE  were  not  useable  and  yielded  bit  error  probabilities 
of  0.5.  The  likely  problem  was  a  lack  of  compatible  RE  interface  between  the  DYSE  and 
the  SUT  which  necessitated  use  of  mixers  and  signal  generators.  However,  this  generated 
additional  noise  and  distorted  the  waveforms.  Thus,  no  data  was  collected  on  the  efficacy 
of  using  the  proposed  test  framework  in  an  emulated  environment. 
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VI.  Conclusion 


Tnis  chapter  first  summarizes  the  eonelusions  drawn  from  the  results  of  researeh  into 
developing  a  test  methodology  to  test  CRSs.  Then,  these  eonelusions  are  used  to 
form  a  basis  for  future  work. 

6.1  Summary 

The  proposed  test  methodology,  or  CRATM,  was  developed  in  response  to  a  perceived 
need  in  the  CR  field  to  have  a  unified  means  of  testing  CR  devices.  The  CR  field  is  diverse 
with  a  wide  range  of  views  on  what  it  means  to  be  a  CR,  what  eognition  is  required,  how 
performance  is  measured,  and  differing  opinions  on  how  to  test.  CRATM  was  developed  in 
response  to  this  lack  of  unification.  CRATM  is  designed  to  be  an  overarehing  framework 
that  allows  CRs  developed  under  different  viewpoints  to  be  tested  using  the  same  test 
methodology. 

CRATM  uses  the  idea  of  benehmarks  to  allow  repeatable,  measurable,  and  compar¬ 
ative  experiments.  CRATM  proposes  to  use  an  emulated  radio  environment,  but  is  not 
required  to  do  so.  With  CRATM,  the  impact  of  the  SU  on  the  PU  can  be  measured  direetly 
by  measuring  the  performance  of  the  PU.  The  PU  ereates  the  radio  environment  that  the  SU 
operates  in.  CRATM  assumes  that  eognition  may  be  measured  by  evaluating  the  ability  of 
the  SU  to  improve  throughput  while  minimizing  interference  to  the  PU  in  accordanee  with 
user  goals  and  poliey.  CR  performance  may  be  measured  by  evaluating  the  performanee  of 
both  the  SU  and  the  performanee  of  the  PU.  This  is  known  as  behavior-based  testing. 

CRATM  was  found  to  be  in  general  agreement  with  the  literature  except  that  it  does 
not  directly  test  eognitive  or  spectrum  sensing  abilities.  In  addition,  CRATM  does  not 
use  psyehometric  or  REM-SDT  testing,  whieh  are  two  proposed  test  methodologies  in  the 
literature.  Behavior-based  testing  was  proposed  as  a  solution  to  these  diserepaneies.  An 
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experiment  was  setup  to  gather  data  on  the  hypothesis  that  CR  cognition  may  be  measured 
by  evaluating  the  performance  of  both  the  SU  and  PU  without  explicitly  evaluating  the 
cognition  of  the  SU. 

The  experiment  did  not  disprove  the  hypothesis  that  SU  cognition  may  be  measured 
via  SU  and  PU  performance.  PU  throughput  was  found  to  increase,  or  stay  the  same, 
for  every  case  where  the  SU  used  the  CE  and/or  a  greater  level  of  cognitive  waveform 
adaptability.  The  SU  throughput  increased  for  PU  narrow -band  environments  but  decreased 
for  PU  wide-band  environments.  However,  this  response  is  to  be  expected  as  the  SU 
adapts  to  the  environment.  BER  of  both  the  PU  and  SU  improved  when  the  SU  CE  was 
turned  on.  Both  PU  and  SU  performance  improved  whenever  the  SU  CE  was  turned  on. 
Additionally,  both  PU  and  SU  performance  improved  or  stayed  the  same  for  greater  levels 
of  SU  cognitive  waveform  adaptability.  Einally,  the  experiment  showed  that  it  is  possible  to 
collect  data  on  SU  and  PU  performance  by  operating  a  SU  and  PU  system  simultaneously. 

Based  on  the  results  of  these  experiments,  behavior-based  testing  was  not  found  to 
be  invalid.  Because  of  this,  CRATM  was  not  shown  to  invalid.  However,  further  research 
is  required  in  order  to  fully  justify  behavior-based  testing  and  CRATM.  A  wider  range  of 
PU  workloads  and  SU  CRSs  need  to  be  evaluated  to  determine  if  behavior-based  testing 
is  effective  across  broad  CR  architectural  features  and  DSA  environments.  Eor  the  limited 
scope  of  this  research,  behavior-based  testing  and  CRATM  was  shown  to  be  effective. 

6.2  Future  Work 

Euture  research  should  continue  to  validate  CRATM  as  well  as  to  continue 
investigation  on  the  efficacy  of  evaluating  SU  and  PU  performance  by  directly  measuring 
their  performance.  Practical  benchmarks  and  benchmark  classes  should  be  formalized  and 
tested  in  accordance  with  the  CRATM  framework.  More  CR  prototypes  with  a  wider 
variety  of  cognition  should  be  tested  under  the  framework,  especially  to  determine  if 
measured  performance  does  in  fact  correlate  with  underlying  cognition.  Euture  research 
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should  transition  the  test  methodology  over  to  the  DYSE  in  order  to  fully  justify  using  an 
emulated  environment  in  CRATM.  In  the  short  term,  ACRO  researchers  should  continue  to 
put  cognitive  functions  onto  hardware  in  order  to  build  and  test  a  stand-alone  CR  prototype. 
There  are  certainly  other  suitable  areas  for  future  research,  but  based  on  the  results  of  this 
research,  these  are  the  most  viable. 
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Appendix  A:  Data  Validation 


This  chapter  analyzes  the  eollected  experimental  data  for  validity.  Besides  the 
throughput  and  BER  data  presented  in  Section  5.1,  additional  experimental  data  was 
collected.  This  additional  data  collected  for  each  experiment  included  the  elapsed 
time,  total  number  of  transmissions,  number  of  symbols  counted,  and  spectrum  sensing 
calibration  data.  The  goal  of  this  data  validation  analysis  is  to  determine  if  appropriate 
experiment  eontrols  were  in  place  and  that  the  eollected  throughput/BER  data  is  valid. 

Eor  the  results  to  be  valid,  there  should  not  be  any  inconsistencies  in  the  data. 
Inconsistencies  are  flagged  if  output  residuals  are  non-normal  or  if  there  are  excessive 
and/or  influential  outliers.  The  measurable  outputs  included  the  total  number  of 
transmissions,  the  number  of  symbols  counted,  the  time  spent  in  the  experiment,  the 
mean  received  signal  strength  (from  ealibration)  as  seen  by  the  spectrum  sensor,  and  the 
computed  threshold  values  for  the  speetrum  sensor.  The  total  number  of  transmissions 
and  the  experiment  elapsed  time  are  impacted  primarily  by  the  the  hardware,  software  and 
experiment  setup.  Inconsistencies  here  indicate  a  problem  in  the  SUT.  The  number  of 
symbols  counted,  as  well  as  the  mean  received  signal  strength  and  thresholds,  are  impacted 
primarily  by  the  external  RE  environment.  Ineonsisteneies  here  indicate  a  problem  in 
the  experimental  test  setup  or  the  SUT.  Throughput  and  BER  data  is  also  examined  for 
problems. 

The  data  is  analyzed  for  two  trends.  Eirst,  each  output  is  examined  for  global  behavior. 
Second,  the  residuals  for  each  output  are  examined  for  each  specific  environment  and  CE 
configuration.  Both  the  global  behavior  and  specifie  configuration  residuals  should  show  a 
normal  distribution  (when  applicable).  If  not,  there  is  a  confounding  factor  that  needs  to  be 
examined. 
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For  the  following  data  analysis,  the  SU  waveform  configurations  will  be  referred  to  as 
a  number.  The  legend  for  the  SU  waveform  configuration  is  shown  in  Table  A.l. 


Table  A.l:  SU  Waveform  Configuration  Legend 


SU  Waveform  Configuration 

Identifier 

FSK,  1  Bins  BW,  CE  Off 

1 

FSK,  8  Bins  BW,  CE  Off 

2 

OEDM,  16  Bins  BW,  CE  Off 

3 

OEDM,  48  Bins  BW,  CE  Off 

4 

ESK,  1  Bins  BW,  CE  On 

5 

ESK,  8  Bins  BW,  CE  On 

6 

OEDM,  16  Bins  BW,  CE  On 

7 

OEDM,  48  Bins  BW,  CE  On 

8 

A.l  Elapsed  Time 

The  first  output  to  be  examined  is  the  time  spent  in  the  experiment,  or  the  elapsed  time. 
This  time  is  measured  from  the  receipt/acknowledgement  of  the  experiment  start  trigger 
signal  until  the  total  experiment  target  time  has  been  reached.  Additionally,  a  time  window 
was  used  to  keep  the  PU  and  SU  transmitting  to  a  semi-synchronous  level.  For  instance, 
if  both  PU  and  SU  are  programmed  for  60  seconds,  it  is  not  desired  to  have  the  SU  finish 
at  60.01  seconds  while  the  PU  starts  a  new  one  second  transmission  at  59.99  seconds.  For 
the  PU,  the  total  experiment  target  time  is  60  seconds  +/-  0.5  seconds.  The  programmed 
duration  for  each  transmit  cycle  is  one  second.  For  the  SU,  the  total  experiment  target  time 
is  60  seconds  +/-  0.125  seconds.  The  programmed  duration  for  each  transmit  cycle  (which 
includes  sensing  the  environment)  is  0.25  seconds.  Of  interest  in  this  analysis  are  any 
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outliers,  and  if  so,  if  they  eome  from  a  partieular  input.  If  any  experiment  eonfigurations 
show  a  eonsistent  bias  in  elapsed  time,  then  the  experiment  results  will  be  impaeted  as  well. 

The  histogram  of  the  elapsed  times  for  both  the  SU  and  PU  are  shown  in  Figure  A.l. 
The  elapsed  times  follow  a  normal  distribution,  as  expeeted.  Next,  the  elapsed  times  are 
investigated  for  non-normal  responses  to  inputs. 


SU  Elapsed  Time  (sec) 
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PU  Elapsed  Time  (sec) 


Figure  A.l:  Histograms  of  Total  Experiment  Elapsed  Time 


Eor  the  SU,  the  input  driver  for  elapsed  time  is  the  ehoiee  of  the  CE.  Eor  the  PU,  the 
eontrolled  input  faetor  for  elapsed  time  is  the  Environment.  In  the  experiment  eode,  the  PU 
environment  is  eonverted  to  a  CE  eonfiguration  with  a  pre-generated  transmit  waveform,  as 
opposed  to  the  SU  whieh  ereates  the  transmit  waveform  based  on  the  sensed  environment. 
The  residuals  for  the  elapsed  time  are  shown  in  Eigure  A. 2.  The  residuals  are  evaluated 
with  respeet  to  the  global  mean  of  elapsed  time. 

In  neither  ease  is  the  eontrolled  input  affeeting  the  elapsed  time  at  a  statistieally 
signifieant  level.  The  residual  outliers  are  likely  due  to  delays  in  eommunieating  with 
the  WARP  boards  when  sending/reeeiving  data  or  eommands.  Regardless  of  the  eause,  the 
data  shows  that  the  SU  and  PU  elapsed  time  follows  the  expeeted  normal  distribution. 
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Figure  A. 2:  Total  Experiment  Elapsed  Time  Residuals 


A.2  Number  of  Transmissions  Received 

The  number  of  transmissions  reeeived  is  the  total  number  of  times  the  receiver  node 
(either  SU  or  PU)  polled  its  RX  buffer  to  capture  the  transmitted  signal.  For  the  PU,  this 
number  is  affected  by  MATLAB  processing  time,  which  is  in  turn  affected  by  the  choice  of 
waveform  modulation.  For  the  SU,  the  number  of  transmissions  received  is  affected  by  the 
use  of  spectrum  sensing  or  not,  the  PU  environment,  and  the  waveform  modulation.  Figure 
A.3  shows  the  relationship  of  the  number  of  transmissions  received  to  the  input  factor  (SU 
waveform  configuration  or  PU  Environment).  As  with  elapsed  time,  the  data  is  examined 
for  outliers  or  data  inconsistencies,  which  may  indicate  problems  either  with  the  SUT. 

As  can  be  seen  in  Figure  A.3,  both  the  PU  and  SU  number  of  transmissions  received 
are  impacted  by  the  choice  of  inputs,  among  other  factors.  The  impact  on  the  PU  is 
discussed  first. 

For  the  PU,  a  linear  model  was  created  to  examine  the  relationship  between  the  number 
of  transmissions  received  and  the  PU  Environment.  Furthermore,  an  additional  factor  was 
added  to  the  model  to  distinguish  between  OFDM  and  FSK  waveforms,  as  well  as  factor  for 
OFDM  modulation/demodulation  time  processing  differences.  While  the  various  OFDM 
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SU  Waveform  Configuration  Environment 

Figure  A. 3:  Scatterplot  of  Total  Number  of  Transmissions  Received 


environments  (Environments  3,  4,  and  5)  execute  the  same  code,  the  bandwidth  changes 
the  number  of  symbols  created  and  modulated,  which  has  an  effect  on  overall  processing 
time.  The  residuals  of  the  fitted  linear  model  are  shown  in  Figure  A. 4  on  both  a  probability 
plot  and  a  histogram. 

The  residuals  approximate  a  normal  distribution;  however,  there  is  a  significant 
leftwards  skew.  Additional  inputs  factors  were  investigated  to  determine  if  they  impacted 
the  PU  number  of  transmissions  received,  such  as  the  order  of  the  trial  to  determine  if  time- 
based  effects  were  present.  None  of  the  additional  input  factors  were  determined  to  have  a 
discernible  effect  on  the  number  of  transmissions  received.  Underlying  causes  for  the  left 
skewed  data  points  could  arise  from  the  limited  amount  of  samples  collected,  the  effect  of 
the  computer  operating  system  and  MATLAB  software,  or  delays  in  communicating  with 
the  WARP  hardware. 

The  SU  number  of  transmissions  show  several  interesting  trends  (see  Figure  A. 3). 
To  begin  with,  the  CE  on  waveform  configurations  (5  -  8)  are  clearly  shifted  downwards 
from  the  CE  off  waveform  configurations.  This  is  indicative  of  the  extra  time  required  to 
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Figure  A.4:  Residuals  of  Fitted  Linear  Model:  PU  Total  Number  of  Transmissions 
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receive  spectrum  data  from  the  WARP  boards  and  process  it.  Furthermore,  evidence  of 
the  CE  waveform  downgrading  to  lower  performing  waveforms  is  clearly  evident  in  the 
dual  clusters  for  waveforms  6-8  (note:  lower  performing  waveforms  actually  have  higher 
number  of  transmissions  received  due  to  less  modulation/demodulation  requirements).  In 
waveform  6,  the  CE  waveform  seems  to  transition  between  FSK-8  down  to  FSK-1  (with 
possibly  2  intermediate  stages  in  between).  The  OFDM  waveforms  show  transitions  from 
the  OFDM  waveform  down  to  FSK  waveforms. 

As  with  the  PU,  a  linear  model  was  fitted  to  the  data  to  ascertain  undesired  behavior. 
Factors  used  to  create  the  linear  model  include  the  PU  environment,  CE  state  (on/off), 
desired  waveform,  and  whether  or  not  the  waveform  is  likely  to  downgrade  based  on  the 
PU  environment.  The  residuals  of  the  model  are  shown  in  Figure  A. 5  for  two  cases.  In 
case  I,  the  data  is  left  unmodified.  In  case  2,  the  outliers,  or  top  clusters  utilizing  FSK 
modulation,  for  waveform  configurations  7  and  8  are  removed. 

The  SU  residuals  prior  to  removing  outliers  show  a  cluster  of  data  not  lying  on  the 
main  regression  line.  Once  the  outliers  are  removed,  the  resulting  residuals  do  show  this 
cluster.  The  data  is  not  completely  normal;  however,  it  is  likely  that  with  an  increased 
sample  size  this  effect  would  go  away. 

A.3  Number  of  Symbols  Counted 

An  output  collected  that  is  similar  to  the  number  of  transmissions  received  is  the 
number  of  symbols  counted.  The  number  of  symbols  counted  is  a  reflection  of  the 
modulation  used  for  the  experiment.  For  the  SU,  the  CE  off  waveform  configuration  should 
show  tight  groupings  whereas  the  CE  on  waveform  configuration  should  show  a  spectrum 
of  discrete  clusters  indicative  of  adaption  to  the  environment  by  changing  the  waveform 
used.  For  the  PU,  the  symbols  counted  should  show  tight  groupings.  Fower  performing 
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Figure  A. 5:  Residuals  of  Fitted  Linear  Model:  SU  Total  Number  of  Transmissions 


waveforms  will  have  a  lower  symbol  eount  than  higher  performing  waveforms.  These 
expeeted  groupings  are  in  faet  found  in  the  data,  as  shown  in  Figure  A. 6. 

A.4  Mean  Received  Signal  Strength  and  Threshold 

A  key  variable  that  will  affeet  SU  and  PU  performanee  is  the  presence  of  unwanted 
RF  interference.  One  way  to  detect  RF  interference  and  to  evaluate  the  stability  of  the  RF 
environment  is  to  chart  the  mean  received  signal  strength  detected  and  thresholds  set  by  the 
SU  during  the  spectrum  sensor  calibration.  The  spectrum  sensor  thresholds  were  calibrated 
prior  to  each  new  experiment.  This  was  done  to  ensure  the  spectrum  sensor  threshold  levels 
were  current  and  accurate.  After  initial  radio  configuration,  the  PU  transmitted  a  known 
waveform.  The  SU,  with  foreknowledge  of  this  known  waveform,  lowered  the  threshold  in 
0.5  dB  increments  until  the  PU  waveform  matched  the  expected  REM.  If  there  are  as  many 
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Figure  A.  6:  Scatterplot  of  Symbols  Counted 


false  REM  values  as  eorreet  REM  values,  the  eurrent  threshold  value  was  thrown  out  and 
the  process  repeated.  Once  a  valid  threshold  was  obtained,  the  process  is  repeated  with  a 
second  known  signal.  Due  to  received  power  fluctuations,  the  SU  was  calibrated  to  the  type 
of  PU  waveform  signal  (ESK  or  OEDM).  The  mean  received  signal  strength  is  the  mean  of 
all  power  spectral  density  bins  computed  using  the  East  Eourier  Transform  (EET). 

The  spectrum  sensor  thresholds  for  both  SU  radios  are  shown  in  Eigure  A.7.  Note  that 
the  threshold  for  Radio  B  is  higher  than  the  threshold  for  Radio  A.  Eor  both,  the  higher 
threshold  grouping  is  the  ESK  threshold,  the  lower  dB  threshold  grouping  is  the  OEDM 
threshold.  The  threshold  values  show  expected  behavior,  though  there  is  one  outlier  at 
-73.5  dB  for  Radio  B. 

The  mean  received  signal  strength  is  shown  plotted  in  Eigure  A. 8.  The  variance 
is  smaller  than  the  threshold  data.  The  mean  received  signal  strength  shows  expected 
behavior. 

Both  the  SU  and  PU  data  for  elapsed  time,  number  of  transmissions  received,  and 
spectrum  sensor  thresholds  do  not  show  substantial  undesired  effects.  Therefore,  there  is 
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Figure  A.7:  Histogram  of  Spectrum  Sensor  Thresholds 
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no  evidence  that  the  SUT  is  not  operating  as  intended,  nor  that  the  external  RF  environment 
is  causing  excessive  variance  in  the  results. 

A.5  Throughput  and  BER 

Next,  the  primary  outputs,  throughput  and  BER,  are  examined.  Boxplots  of  the  raw 
data  are  shown  in  Figure  A. 9.  A  linear  model  was  fitted  to  the  data.  The  predictors 
used  in  the  linear  model  included  the  PU  Environment,  SU  desired  waveform,  CE  status 
(on/off),  experimental  order,  spectrum  sensor  thresholds,  and  elapsed  times.  The  linear 
model  predictors  and  statistics  are  shown  in  Table  A. 2.  Figure  A.  10  shows  the  linear  model 
fitted  residuals  for  throughput  and  BER  for  both  the  SU  and  PU. 


Table  A. 2:  Fitted  Einear  Model  Parameters 


Predictor 

Estimate 

Standard  Error 

t- statistic 

p-value 

(Intercept) 

-1.1804 

0.90953 

-1.2979 

0.19461 

ExpNum 

-1.6098e-05 

7.5581e-06 

-2.1299 

0.033406 

ExpConfig 

-0.0043267 

0.00014747 

-29.339 

1.0238e-139 

ElapsedTimeP 

0.0094172 

0.0065644 

1.4346 

0.15169 

ElapsedTimeS 

0.0088554 

0.013314 

0.66512 

0.50612 

WaveformP 

-0.013832 

0.062051 

-0.22292 

0.82364 

Environment 

0.032127 

0.002802 

11.466 

8.3421e-29 

CogEngine 

0 

0 

NaN 

NaN 

Thresh  A 

0.00018669 

0.0028943 

0.064504 

0.94858 

ThreshB 

-0.0020201 

0.0029318 

-0.68902 

0.49096 

EnvFactor 

0.037239 

0.0031466 

11.835 

1.7814e-30 

WE 

0.034152 

0.001789 

19.089 

2.9834e-70 
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Figure  A. 9:  Experimental  Data  for  Throughput  and  BER 
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Figure  A.  10:  Experimental  Data  for  Throughput  and  BER 
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The  residuals  generally  follow  the  normal  distribution.  All  residuals  show  non-ideal 
distributions  at  the  tails.  This  could  reflect  either  a  unknown  factor,  a  shortage  of  sample 
data,  or  the  results  of  attempting  to  use  a  linear  model  with  non-linear  predictors.  Predictors 
were  added  to  best  capture  the  non-linear  relationship  in  the  SU  CE  and  PU  Environments. 
Eor  example,  the  SU  OEDM-48  behavior  with  the  CE  on  may  vary  considerably  as  shown 
in  Eigure  A.9.  The  relationship  between  the  output  and  the  SU  waveform  configuration  is 
seen  to  be  non-linear.  To  remedy  this,  additional  predictors  were  used  to  clarify  the  CE  on 
behavior. 
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Appendix  B:  Spectrum  Sensor  Core 


N  energy-detection  based  spectrum  sensor  was  developed  for  use  in  this  research 


1.  effort.  The  spectrum  sensor  was  designed  to  be  incorporated  into  the  WARP  boards 
as  an  IP  core.  The  spectrum  sensor  may  be  used  on  any  Xilinx  FPGA  with  sufficient  area. 
This  section  describes  the  spectrum  sensor  developed  for  this  and  future  ACRO  research. 

B.l  Background  on  Energy  Detection  for  Spectrum  Sensing 

The  principle  of  energy  detection  is  discussed  further  in  [27,  57].  Energy  detection 
is  the  optimal  method  when  only  power  measurements  are  available  [57,  58]  and  is 
easy  to  design  and  implement  [27].  In  its  simplest  form,  energy  detection  categorizes 
measurements  above  a  certain  threshold  as  occupied  spectrum  as  shown  in  Figure  4.2. 
When  noise  statistics  are  known,  three  primary  means  of  detecting  signals  are  the  m-dB 
criterion,  maximum  noise  level  criterion,  and  the  Probability  of  False  Alarm  (PFA)  criterion 
[59].  In  the  m-dB  criterion,  the  threshold  is  simply  set  at  an  arbitrary  value  above  the  noise 
floor  such  as  6  or  10  dB.  In  the  maximum  noise  level  criterion,  the  threshold  is  set  at 
the  maximum  noise  level.  In  the  PFA  criterion,  the  threshold  is  set  at  a  level  such  that 
the  threshold  leads  to  a  specified  probability  of  false  alarm.  This  is  discussed  further  in 
[27,  60].  There  are  algorithms  that  do  not  need  prior  knowledge  of  noise  properties,  these 
are  Otsu’s  algorithm  and  the  Recursive  One-Side  Hypothesis  Testing  algorithm  [61,  62], 
though  these  are  outside  the  scope  of  the  spectrum  sensor  for  this  research. 

The  spectrum  sensor  was  designed  to  apply  a  threshold  to  the  incoming  data  to 
determine  if  spectrum  is  occupied  or  not.  Furthermore,  the  spectrum  sensor  was  designed 
to  accommodate  all  three  energy  detection  schemes  presented  above.  That  is,  the  spectrum 
sensor  has  both  a  changeable  threshold  in  addition  to  providing  noise  statistics  (average 
noise  floor  and  variance).  Each  EFT  output  bin  corresponds  to  a  spectrum  bin.  In  the  output 
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REM,  each  bit  is  a  one  or  zero  as  it  relates  to  the  threshold  level  for  the  corresponding 
frequency.  Zeros  indicate  an  open  frequency;  ones  indicate  an  occupied  frequency.  The 
functional  block  diagram  of  the  spectrum  sensor  is  shown  in  Figure  B.l.  The  variable  n 
corresponds  to  the  number  of  frequency  bins  to  be  used. 


spectmm_sensor 


Figure  B.l:  Spectrum  Sensor  Functional  Block  Diagram 


B.2  Spectrum  Sensor  Design  Overview 

The  spectrum  sensor  was  designed  to  accommodate  a  range  of  spectrum  sensing 
needs.  To  do  this,  the  underlying  code  was  built  to  respond  to  a  flexible  range  of  inputs. 
The  spectrum  sensor  has  the  following  characteristics: 

•  Reconfigurable  FFT  Size:  The  FFT  size  is  configurable  to  sizes  from  2®  to  2^^.  All 
FFT  sizes  are  powers  of  2.  The  FFT  size  may  be  set  by  setting  the  FFT  write  enable 
(NFFT.WE)  flag  high. 
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•  14-Bit  I/Q  Inputs:  The  input  accepts  up  to  14  bit  signed  integers  for  both  I  and  Q 
components. 

•  Reconfigurable  Threshold  Level:  The  threshold  level  may  be  set  according  to  user 
needs  anytime  during  operation.  The  threshold  level  is  an  unsigned  32-bit  integer. 

•  Variable  Size  REM  The  REM  output  is  of  variable  length  from  32  bits  to  1024  bits,  in 
32  bit- increments.  That  is,  there  are  thirty-two  32-bit  output  registers  that  correspond 
to  the  output  REM. 

•  Variable  Selection  of  Spectrum  Bins:  The  user  has  control  over  which  EET-output 
bins  are  used  to  create  the  REM.  Eor  example,  a  user  may  be  interested  in  only  64 
particular  frequencies/bins,  though  a  1024-point  EET  is  used.  The  user  may  specify 
the  frequency/bin  range  of  interest  in  contiguous  32-bit  chunks. 

•  Spectrum  Bin  Masking  (Hopset  Mask):  The  user  may  specify  certain  frequency  bins 
to  be  excluded  from  the  REM.  This  is  useful  for  frequency  hopping  applications 
when  spectrum  sensing  occurs  simultaneously  with  user  transmissions. 

•  Noise  Floor  Average  Value:  The  average  noise  floor,  or  mean  received  signal 
strength,  is  computed  using  the  power  spectral  density  values.  This  is  output  as  a 
32-bit  integer. 

•  Noise  Floor  Variance:  The  variance  of  the  noise  floor,  or  received  signal,  is  computed 
and  output  as  a  32-bit  integer. 

•  Data  Valid  Check:  The  user  has  control  over  the  spectrum  sensor  start/stop  flag. 
Eurthermore,  the  user  can  monitor  the  output  data  valid  flags  such  that  when  a  flag  is 
set  to  high,  the  data  is  ready  to  be  read.  This  prevents  the  user  from  using  obsolete 
or  redundant  data,  should  the  spectrum  sensor  start  flag  be  set  to  low  or  if  the  EET 
determines  the  output  is  invalid. 
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These  eharaeteristics  may  be  tied  together  in  an  illustration.  A  user  first  speeifies  the 
desired  FFT  size,  which  for  this  example  if  2^  =  256.  The  user  desires  to  use  frequency 
bins  2  to  66,  which  correspond  to  a  frequency  range  of  0.3125  to  10.3125  MHz.  The  REM 
size  is  n  =  64.  Frequency  bins  outside  of  the  range  2  to  66  are  not  included  in  the  REM, 
noise,  or  variance  calculations.  Furthermore,  the  user  does  not  seek  to  use  frequency  bin 
35  due  to  known  interference  issues;  therefore,  the  hopset  mask  is  set  to  discard  bin  35. 

The  schematic  of  the  spectrum  sensor  is  shown  in  Figure  B.2.  Sub-components  of  the 
spectrum  sensor  correspond  to  the  dashed  line  blocks  in  Figure  B.l.  The  FFT  core  used 
in  the  spectrum  sensor  is  the  Xilinx  DS260  Fast  Fourier  Transform  v7.1.  Other  IP  cores 
include  a  RAM  block  (Xilinx  DS512)  in  noise_compute  and  multipliers  (Xilinx  DS255) 
in  psd_block. 

The  inputs  of  the  spectrum  sensor  are  as  follows.  xn_re  and  xn.im  refer  to  the  I  and 
Q  components  of  the  input  signal.  The  threshold  and  ££t_size  are  self-explanatory. 
n££t  we  is  used  as  a  signal  to  write  the  ££t  size,  nuiti  rems  is  the  number  of  REMs  to 
be  output,  where  each  REM  is  32  bits.  Due  to  implementation  constraints,  the  number 
of  REMs  must  be  1,  2,  4,  8,  16,  or  32.  The  bwjtiaskjtnode  is  used  to  set  the  indexing 
scheme  used  to  output  the  REMs.  There  are  three  modes.  Mode  1  uses  all  FFT  outputs 
(only  valid  for  NFFT  <=  1024).  Mode  2  outputs  only  positive  frequency  components 
(only  valid  for  NFFT  <=  2048).  Mode  3  allows  mixed  positive  and  negative  frequency 
components,  which  are  specified  using  the  next  four  inputs.  The  bwjnask  components 
correspond  to  the  minimum  positive  frequency  index,  maximum  positive  frequency  index, 
minimum  negative  frequency  index,  and  maximum  frequency  index.  The  start  input  is 
used  to  trigger  the  operation  of  the  spectrum  sensor,  start  is  tied  high  for  continuous 
operation.  Finally,  there  are  thirty-two  32-bit  hopset  jnask  registers  which  correspond  to 
desired  REM  output  frequency  bins. 
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Figure  B.2:  Spectrum  Sensor  Top-Level  Schematic 
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The  outputs  of  the  spectrum  sensor  are  the  average  noise  (noise.avg),  variance 
(var  regO  and  var  regl)  ,  and  thirty-two  32-bit  REM  registers.  Each  output  also  has 
a  valid  flag  which  goes  high  one  clock-cycle  prior  to  the  respective  output  becoming  valid. 
Eor  example,  in  streaming  operation  with  NFFT  =  256,  the  rem.valid  flag  will  go  high 
every  256  clock  cycles. 

B.3  Performance,  Area,  and  Timing 

System  performance  of  the  spectrum  sensor  was  validated  using  a  Xilinx  System 
Generator  testbench.  In  this  testbench,  a  MATEAB  script  created  the  range  of  inputs, 
which  were  then  executed  in  the  System  Generator  testbench  module,  and  the  outputs  were 
validated  against  predicted  responses  in  MATEAB.  Eollowing  this,  system  performance 
was  validated  on  the  WARP  board  for  a  special  case  of  NFFT  =  256  with  frequency  bins 
[1  64]  to  create  two  32-bit  REMs.  The  spectrum  sensor  was  found  to  exhibit  the  desired 
outputs  for  the  REM,  noise,  and  variance  by  comparing  the  results  with  stored  I/Q  samples 
pulled  from  the  board  using  WARPEab. 

The  spectrum  sensor  was  designed  to  use  minimum  area  while  preserving  perfor¬ 
mance.  The  Xilinx  EET  core  uses  a  relatively  large  amount  of  DSP48s,  which  allow  fast 
mathematical  operations.  To  implement  the  design  in  WARPEab,  it  was  not  necessary  to 
change  EET  core  instance.  However,  it  was  necessary  to  change  the  EET  core  to  use  logic 
slices  instead  of  DSP48s  for  implementation  on  other  WARP  designs  such  as  the  WARP 
OEDM  reference  design.  Performance  and  timing  were  not  affected  by  this  change.  Eor 
future  use,  the  EET  core  may  be  setup  for  a  specific  EET  size  in  order  to  save  space. 

The  area  summary  for  the  stock  spectrum  sensor  (reconfigurable  EET  up  to  NFFT  = 
16384)  is  shown  in  Table  B.l.  The  synthesis  was  targeted  towards  the  Virtex  4  chip 
(xc4vfxl 00-1 1111517)  used  on  the  WARP  v2  board  using  a  balanced  synthesis  goal.  The 
synthesis  engine  was  the  default  choice  in  Xilinx  Integrated  Software  Environment  (ISE) 
13.2. 
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Table  B.l:  Spectrum  Sensor  Area  Summary 


Eogic  Utilization 

Used 

Available 

Utilization 

Number  of  Slices 

4819 

42176 

11% 

Number  of  Slice  Elip  Elops 

6397 

84352 

7% 

Number  of  4-input  EUTs 

5954 

84352 

7% 

Number  of  EIEO 1 6/RAMB 1 6s 

4 

376 

1% 

Number  of  DSP48s 

31 

160 

19% 

The  total  duration  is  proportional  to  NFFT .  Each  sub-component  in  the  spectrum 
sensor  also  takes  a  time  proportional  to  NFFT  to  compute  its  output.  In  addition,  the 
number  of  REMs  also  impacts  the  total  duration.  Eor  example,  for  a  NFFT  =  1024  case 
with  only  one  32-bit  REM,  the  total  duration  will  be  1024  -  32  =  992  clock  cycles  shorter 
than  if  all  32  REMs  were  used.  Table  B.2  shows  the  time  from  start  going  high  or  the  first 
valid  input  until  the  corresponding  first  valid  output  flag  for  the  REM,  noise  average,  and 
variance. 

The  spectrum  sensor  is  designed  to  be  clocked  to  the  I/Q  sample  rate.  Maximum  clock 
rate  is  64.4  MHz.  This  relatively  slow  clock  rate  may  be  increased  by  fixing  one  particular 
logic  sequence  in  the  EET  core;  this  was  not  done  for  this  research  as  the  clock  rate  is 
sufficient  to  be  used  on  the  WARP  core.  As  originally  designed,  the  spectrum  sensor  was 
intended  for  use  with  small  NFFT  sizes,  which  makes  it  feasible  to  operate  at  the  I/Q 
sample  rate.  However,  future  revisions  of  the  spectrum  sensor  should  investigate  buffering 
the  I/Q  inputs  and  then  operating  the  spectrum  sensor  at  a  much  higher  clock  rate  in  order 
to  shorten  the  overall  latency. 
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Table  B.2:  Spectrum  Sensor  Timing  Summary 


REM  Output  Ready  Noise  Avg  Output  Ready  Variance  Output  Ready 


NFFT 

Clock  Cycles 

Time  (jus) 

Clock  Cycles 

Time  (jus) 

Clock  Cycles 

Time  (jus) 

64 

288 

7.2 

286 

7.15 

354 

8.85 

128 

502 

12.55 

500 

12.5 

632 

15.8 

256 

885 

22.125 

883 

22.075 

1143 

28.575 

512 

1675 

41.875 

1673 

41.825 

2189 

54.725 

1024 

3210 

80.25 

3208 

80.2 

4236 

105.9 

2048 

6304 

157.6 

6302 

157.55 

7330 

183.25 

4096 

12447 

311.175 

12445 

311.125 

13473 

336.825 

8192 

24756 

618.9 

24754 

618.85 

25782 

644.55 

16384 

49331 

1233.275 

49329 

1233.225 

50357 

1258.925 

B.4  WARP  Implementation 

The  spectrum  sensor  was  successfully  integrated  with  the  WARP  v2  board.  The 
spectrum  sensor  I/Q  inputs  were  tied  directly  to  the  radio  bridge  outputs.  The  remaining 
spectrum  sensor  inputs  and  outputs  were  tied  to  the  40  MHz  PLB.  The  embedded  PowerPC 
405  chip  on  the  Virtex  4  ran  the  software  necessary  to  operate  the  spectrum  sensor  inputs 
and  outputs.  Besides  the  C  software  written  to  integrate  the  spectrum  sensor  with  the 
board,  additional  software  was  written  in  MATLAB  such  that  the  WARPLab  software  could 
control  the  spectrum  sensor  from  the  PC.  The  user  can  configure  the  spectrum  sensor  using 
MATLAB,  as  well  as  retrieve  the  REMs,  noise,  and  variance  values. 
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